{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from mpl_toolkits import mplot3d # 三维画图\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "import sys\n",
    "sys.path.append(\"..\") \n",
    "import dl_utils\n",
    "import torch\n",
    "import torch.optim as optim\n",
    "import torch.nn as nn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimizer\n",
    "### 在一个深度学习问题中，我们通常会预先定义一个损失函数。\n",
    "\n",
    "### 有了损失函数以后，我们就可以使用优化算法试图将其最小化。\n",
    "\n",
    "在优化中，这样的损失函数通常被称作优化问题的目标函数（objective function）。依据惯例，优化算法通常只考虑最小化目标函数。\n",
    "\n",
    "这里不考虑 泛化误差和训练误差，只考虑 最小化目标函数的问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "深度学习中绝大多数目标函数都很复杂。因此，很多优化问题并不存在解析解，而需要使用基于数值方法的优化算法找到近似解，即数值解。本书中讨论的优化算法都是这类基于数值方法的算法。为了求得最小化目标函数的数值解，我们将通过优化算法有限次迭代模型参数来尽可能降低损失函数的值。\n",
    "\n",
    "优化在深度学习中有很多挑战。下面描述了其中的两个挑战，即局部最小值和鞍点。为了更好地描述问题，我们先导入本节中实验需要的包或模块。\n",
    "\n",
    "## 局部最小值\n",
    "\n",
    "对于目标函数$f(x)$，如果$f(x)$在$x$上的值比在$x$邻近的其他点的值更小，那么$f(x)$可能是一个局部最小值（local minimum）。如果$f(x)$在$x$上的值是目标函数在整个定义域上的最小值，那么$f(x)$是全局最小值（global minimum）。\n",
    "\n",
    "举个例子，给定函数\n",
    "$$\n",
    "f(x) = x \\cdot \\text{cos}(\\pi x), \\qquad -1.0 \\leq x \\leq 2.0,\n",
    "$$\n",
    "我们可以大致找出该函数的局部最小值和全局最小值的位置。需要注意的是，图中箭头所指示的只是大致位置。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\r\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 305.329712 184.15625\" width=\"305.329712pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 184.15625 \r\n",
       "L 305.329712 184.15625 \r\n",
       "L 305.329712 -0 \r\n",
       "L 0 -0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 42.620313 146.6 \r\n",
       "L 293.720313 146.6 \r\n",
       "L 293.720313 10.7 \r\n",
       "L 42.620313 10.7 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m48f1e6e62c\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"54.033949\" xlink:href=\"#m48f1e6e62c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- −1.0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\"/>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\"/>\r\n",
       "       <path d=\"M 10.6875 12.40625 \r\n",
       "L 21 12.40625 \r\n",
       "L 21 0 \r\n",
       "L 10.6875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2e\"/>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(41.892543 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"93.391316\" xlink:href=\"#m48f1e6e62c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- −0.5 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.796875 72.90625 \r\n",
       "L 49.515625 72.90625 \r\n",
       "L 49.515625 64.59375 \r\n",
       "L 19.828125 64.59375 \r\n",
       "L 19.828125 46.734375 \r\n",
       "Q 21.96875 47.46875 24.109375 47.828125 \r\n",
       "Q 26.265625 48.1875 28.421875 48.1875 \r\n",
       "Q 40.625 48.1875 47.75 41.5 \r\n",
       "Q 54.890625 34.8125 54.890625 23.390625 \r\n",
       "Q 54.890625 11.625 47.5625 5.09375 \r\n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \r\n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \r\n",
       "Q 12.796875 0.140625 7.71875 1.703125 \r\n",
       "L 7.71875 11.625 \r\n",
       "Q 12.109375 9.234375 16.796875 8.0625 \r\n",
       "Q 21.484375 6.890625 26.703125 6.890625 \r\n",
       "Q 35.15625 6.890625 40.078125 11.328125 \r\n",
       "Q 45.015625 15.765625 45.015625 23.390625 \r\n",
       "Q 45.015625 31 40.078125 35.4375 \r\n",
       "Q 35.15625 39.890625 26.703125 39.890625 \r\n",
       "Q 22.75 39.890625 18.8125 39.015625 \r\n",
       "Q 14.890625 38.140625 10.796875 36.28125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-35\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(81.249909 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"132.748682\" xlink:href=\"#m48f1e6e62c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 0.0 -->\r\n",
       "      <g transform=\"translate(124.79712 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"172.106049\" xlink:href=\"#m48f1e6e62c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 0.5 -->\r\n",
       "      <g transform=\"translate(164.154487 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"211.463416\" xlink:href=\"#m48f1e6e62c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 1.0 -->\r\n",
       "      <g transform=\"translate(203.511853 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"250.820783\" xlink:href=\"#m48f1e6e62c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 1.5 -->\r\n",
       "      <g transform=\"translate(242.86922 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_7\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"290.178149\" xlink:href=\"#m48f1e6e62c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 2.0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(282.226587 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_8\">\r\n",
       "     <!-- x -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 54.890625 54.6875 \r\n",
       "L 35.109375 28.078125 \r\n",
       "L 55.90625 0 \r\n",
       "L 45.3125 0 \r\n",
       "L 29.390625 21.484375 \r\n",
       "L 13.484375 0 \r\n",
       "L 2.875 0 \r\n",
       "L 24.125 28.609375 \r\n",
       "L 4.6875 54.6875 \r\n",
       "L 15.28125 54.6875 \r\n",
       "L 29.78125 35.203125 \r\n",
       "L 44.28125 54.6875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(165.210938 174.876562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m34e308bef6\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620313\" xlink:href=\"#m34e308bef6\" y=\"138.423853\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- −1 -->\r\n",
       "      <g transform=\"translate(20.878125 142.223072)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620313\" xlink:href=\"#m34e308bef6\" y=\"95.122727\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(29.257813 98.921946)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620313\" xlink:href=\"#m34e308bef6\" y=\"51.821601\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- 1 -->\r\n",
       "      <g transform=\"translate(29.257813 55.62082)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_12\">\r\n",
       "     <!-- f(x) -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 37.109375 75.984375 \r\n",
       "L 37.109375 68.5 \r\n",
       "L 28.515625 68.5 \r\n",
       "Q 23.6875 68.5 21.796875 66.546875 \r\n",
       "Q 19.921875 64.59375 19.921875 59.515625 \r\n",
       "L 19.921875 54.6875 \r\n",
       "L 34.71875 54.6875 \r\n",
       "L 34.71875 47.703125 \r\n",
       "L 19.921875 47.703125 \r\n",
       "L 19.921875 0 \r\n",
       "L 10.890625 0 \r\n",
       "L 10.890625 47.703125 \r\n",
       "L 2.296875 47.703125 \r\n",
       "L 2.296875 54.6875 \r\n",
       "L 10.890625 54.6875 \r\n",
       "L 10.890625 58.5 \r\n",
       "Q 10.890625 67.625 15.140625 71.796875 \r\n",
       "Q 19.390625 75.984375 28.609375 75.984375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-66\"/>\r\n",
       "      <path d=\"M 31 75.875 \r\n",
       "Q 24.46875 64.65625 21.28125 53.65625 \r\n",
       "Q 18.109375 42.671875 18.109375 31.390625 \r\n",
       "Q 18.109375 20.125 21.3125 9.0625 \r\n",
       "Q 24.515625 -2 31 -13.1875 \r\n",
       "L 23.1875 -13.1875 \r\n",
       "Q 15.875 -1.703125 12.234375 9.375 \r\n",
       "Q 8.59375 20.453125 8.59375 31.390625 \r\n",
       "Q 8.59375 42.28125 12.203125 53.3125 \r\n",
       "Q 15.828125 64.359375 23.1875 75.875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-28\"/>\r\n",
       "      <path d=\"M 8.015625 75.875 \r\n",
       "L 15.828125 75.875 \r\n",
       "Q 23.140625 64.359375 26.78125 53.3125 \r\n",
       "Q 30.421875 42.28125 30.421875 31.390625 \r\n",
       "Q 30.421875 20.453125 26.78125 9.375 \r\n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \r\n",
       "L 8.015625 -13.1875 \r\n",
       "Q 14.5 -2 17.703125 9.0625 \r\n",
       "Q 20.90625 20.125 20.90625 31.390625 \r\n",
       "Q 20.90625 42.671875 17.703125 53.65625 \r\n",
       "Q 14.5 64.65625 8.015625 75.875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-29\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798438 87.271094)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\r\n",
       "      <use x=\"35.205078\" xlink:href=\"#DejaVuSans-28\"/>\r\n",
       "      <use x=\"74.21875\" xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "      <use x=\"133.398438\" xlink:href=\"#DejaVuSans-29\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_11\">\r\n",
       "    <path clip-path=\"url(#pf01846f6f1)\" d=\"M 54.033949 51.821601 \r\n",
       "L 61.905422 58.059091 \r\n",
       "L 69.776896 67.09765 \r\n",
       "L 77.648369 77.306493 \r\n",
       "L 85.519842 87.094257 \r\n",
       "L 93.391316 95.122727 \r\n",
       "L 101.262789 100.475041 \r\n",
       "L 109.134262 102.758256 \r\n",
       "L 117.005736 102.128997 \r\n",
       "L 124.877209 99.240909 \r\n",
       "L 132.748682 95.122727 \r\n",
       "L 140.620156 91.004545 \r\n",
       "L 148.491629 88.116458 \r\n",
       "L 156.363102 87.487198 \r\n",
       "L 164.234576 89.770414 \r\n",
       "L 172.106049 95.122727 \r\n",
       "L 179.977523 103.151198 \r\n",
       "L 187.848996 112.938962 \r\n",
       "L 195.720469 123.147805 \r\n",
       "L 203.591943 132.186364 \r\n",
       "L 211.463416 138.423853 \r\n",
       "L 219.334889 140.422727 \r\n",
       "L 227.206363 137.160344 \r\n",
       "L 235.077836 128.21002 \r\n",
       "L 242.949309 113.855825 \r\n",
       "L 250.820783 95.122727 \r\n",
       "L 258.692256 73.713473 \r\n",
       "L 266.563729 51.85473 \r\n",
       "L 274.435203 32.066303 \r\n",
       "L 282.306676 16.877273 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 42.620313 146.6 \r\n",
       "L 42.620313 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 293.720313 146.6 \r\n",
       "L 293.720313 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 42.620313 146.6 \r\n",
       "L 293.720313 146.6 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 42.620313 10.7 \r\n",
       "L 293.720313 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_7\">\r\n",
       "    <path d=\"M 109.113525 126.824204 \r\n",
       "Q 109.1229 117.386339 109.131165 109.066508 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\r\n",
       "    <path d=\"M 107.127192 113.064519 \r\n",
       "L 109.131165 109.066508 \r\n",
       "L 111.12719 113.068492 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"text_13\">\r\n",
       "    <!-- local minimum -->\r\n",
       "    <defs>\r\n",
       "     <path d=\"M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6c\"/>\r\n",
       "     <path d=\"M 30.609375 48.390625 \r\n",
       "Q 23.390625 48.390625 19.1875 42.75 \r\n",
       "Q 14.984375 37.109375 14.984375 27.296875 \r\n",
       "Q 14.984375 17.484375 19.15625 11.84375 \r\n",
       "Q 23.34375 6.203125 30.609375 6.203125 \r\n",
       "Q 37.796875 6.203125 41.984375 11.859375 \r\n",
       "Q 46.1875 17.53125 46.1875 27.296875 \r\n",
       "Q 46.1875 37.015625 41.984375 42.703125 \r\n",
       "Q 37.796875 48.390625 30.609375 48.390625 \r\n",
       "z\r\n",
       "M 30.609375 56 \r\n",
       "Q 42.328125 56 49.015625 48.375 \r\n",
       "Q 55.71875 40.765625 55.71875 27.296875 \r\n",
       "Q 55.71875 13.875 49.015625 6.21875 \r\n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \r\n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \r\n",
       "Q 5.515625 13.875 5.515625 27.296875 \r\n",
       "Q 5.515625 40.765625 12.171875 48.375 \r\n",
       "Q 18.84375 56 30.609375 56 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6f\"/>\r\n",
       "     <path d=\"M 48.78125 52.59375 \r\n",
       "L 48.78125 44.1875 \r\n",
       "Q 44.96875 46.296875 41.140625 47.34375 \r\n",
       "Q 37.3125 48.390625 33.40625 48.390625 \r\n",
       "Q 24.65625 48.390625 19.8125 42.84375 \r\n",
       "Q 14.984375 37.3125 14.984375 27.296875 \r\n",
       "Q 14.984375 17.28125 19.8125 11.734375 \r\n",
       "Q 24.65625 6.203125 33.40625 6.203125 \r\n",
       "Q 37.3125 6.203125 41.140625 7.25 \r\n",
       "Q 44.96875 8.296875 48.78125 10.40625 \r\n",
       "L 48.78125 2.09375 \r\n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \r\n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \r\n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \r\n",
       "Q 5.515625 14.109375 5.515625 27.296875 \r\n",
       "Q 5.515625 40.671875 12.859375 48.328125 \r\n",
       "Q 20.21875 56 33.015625 56 \r\n",
       "Q 37.15625 56 41.109375 55.140625 \r\n",
       "Q 45.0625 54.296875 48.78125 52.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-63\"/>\r\n",
       "     <path d=\"M 34.28125 27.484375 \r\n",
       "Q 23.390625 27.484375 19.1875 25 \r\n",
       "Q 14.984375 22.515625 14.984375 16.5 \r\n",
       "Q 14.984375 11.71875 18.140625 8.90625 \r\n",
       "Q 21.296875 6.109375 26.703125 6.109375 \r\n",
       "Q 34.1875 6.109375 38.703125 11.40625 \r\n",
       "Q 43.21875 16.703125 43.21875 25.484375 \r\n",
       "L 43.21875 27.484375 \r\n",
       "z\r\n",
       "M 52.203125 31.203125 \r\n",
       "L 52.203125 0 \r\n",
       "L 43.21875 0 \r\n",
       "L 43.21875 8.296875 \r\n",
       "Q 40.140625 3.328125 35.546875 0.953125 \r\n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \r\n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \r\n",
       "Q 6 8.015625 6 15.921875 \r\n",
       "Q 6 25.140625 12.171875 29.828125 \r\n",
       "Q 18.359375 34.515625 30.609375 34.515625 \r\n",
       "L 43.21875 34.515625 \r\n",
       "L 43.21875 35.40625 \r\n",
       "Q 43.21875 41.609375 39.140625 45 \r\n",
       "Q 35.0625 48.390625 27.6875 48.390625 \r\n",
       "Q 23 48.390625 18.546875 47.265625 \r\n",
       "Q 14.109375 46.140625 10.015625 43.890625 \r\n",
       "L 10.015625 52.203125 \r\n",
       "Q 14.9375 54.109375 19.578125 55.046875 \r\n",
       "Q 24.21875 56 28.609375 56 \r\n",
       "Q 40.484375 56 46.34375 49.84375 \r\n",
       "Q 52.203125 43.703125 52.203125 31.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-61\"/>\r\n",
       "     <path id=\"DejaVuSans-20\"/>\r\n",
       "     <path d=\"M 52 44.1875 \r\n",
       "Q 55.375 50.25 60.0625 53.125 \r\n",
       "Q 64.75 56 71.09375 56 \r\n",
       "Q 79.640625 56 84.28125 50.015625 \r\n",
       "Q 88.921875 44.046875 88.921875 33.015625 \r\n",
       "L 88.921875 0 \r\n",
       "L 79.890625 0 \r\n",
       "L 79.890625 32.71875 \r\n",
       "Q 79.890625 40.578125 77.09375 44.375 \r\n",
       "Q 74.3125 48.1875 68.609375 48.1875 \r\n",
       "Q 61.625 48.1875 57.5625 43.546875 \r\n",
       "Q 53.515625 38.921875 53.515625 30.90625 \r\n",
       "L 53.515625 0 \r\n",
       "L 44.484375 0 \r\n",
       "L 44.484375 32.71875 \r\n",
       "Q 44.484375 40.625 41.703125 44.40625 \r\n",
       "Q 38.921875 48.1875 33.109375 48.1875 \r\n",
       "Q 26.21875 48.1875 22.15625 43.53125 \r\n",
       "Q 18.109375 38.875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.1875 51.21875 25.484375 53.609375 \r\n",
       "Q 29.78125 56 35.6875 56 \r\n",
       "Q 41.65625 56 45.828125 52.96875 \r\n",
       "Q 50 49.953125 52 44.1875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6d\"/>\r\n",
       "     <path d=\"M 9.421875 54.6875 \r\n",
       "L 18.40625 54.6875 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 64.59375 \r\n",
       "L 9.421875 64.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-69\"/>\r\n",
       "     <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6e\"/>\r\n",
       "     <path d=\"M 8.5 21.578125 \r\n",
       "L 8.5 54.6875 \r\n",
       "L 17.484375 54.6875 \r\n",
       "L 17.484375 21.921875 \r\n",
       "Q 17.484375 14.15625 20.5 10.265625 \r\n",
       "Q 23.53125 6.390625 29.59375 6.390625 \r\n",
       "Q 36.859375 6.390625 41.078125 11.03125 \r\n",
       "Q 45.3125 15.671875 45.3125 23.6875 \r\n",
       "L 45.3125 54.6875 \r\n",
       "L 54.296875 54.6875 \r\n",
       "L 54.296875 0 \r\n",
       "L 45.3125 0 \r\n",
       "L 45.3125 8.40625 \r\n",
       "Q 42.046875 3.421875 37.71875 1 \r\n",
       "Q 33.40625 -1.421875 27.6875 -1.421875 \r\n",
       "Q 18.265625 -1.421875 13.375 4.4375 \r\n",
       "Q 8.5 10.296875 8.5 21.578125 \r\n",
       "z\r\n",
       "M 31.109375 56 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-75\"/>\r\n",
       "    </defs>\r\n",
       "    <g transform=\"translate(72.138338 138.423853)scale(0.1 -0.1)\">\r\n",
       "     <use xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "     <use x=\"27.783203\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "     <use x=\"88.964844\" xlink:href=\"#DejaVuSans-63\"/>\r\n",
       "     <use x=\"143.945312\" xlink:href=\"#DejaVuSans-61\"/>\r\n",
       "     <use x=\"205.224609\" xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "     <use x=\"233.007812\" xlink:href=\"#DejaVuSans-20\"/>\r\n",
       "     <use x=\"264.794922\" xlink:href=\"#DejaVuSans-6d\"/>\r\n",
       "     <use x=\"362.207031\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "     <use x=\"389.990234\" xlink:href=\"#DejaVuSans-6e\"/>\r\n",
       "     <use x=\"453.369141\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "     <use x=\"481.152344\" xlink:href=\"#DejaVuSans-6d\"/>\r\n",
       "     <use x=\"578.564453\" xlink:href=\"#DejaVuSans-75\"/>\r\n",
       "     <use x=\"641.943359\" xlink:href=\"#DejaVuSans-6d\"/>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_8\">\r\n",
       "    <path d=\"M 220.407145 66.562567 \r\n",
       "Q 219.88643 100.408884 219.382913 133.137299 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\r\n",
       "    <path d=\"M 221.444207 129.168538 \r\n",
       "L 219.382913 133.137299 \r\n",
       "L 217.444681 129.107006 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"text_14\">\r\n",
       "    <!-- global minimum -->\r\n",
       "    <defs>\r\n",
       "     <path d=\"M 45.40625 27.984375 \r\n",
       "Q 45.40625 37.75 41.375 43.109375 \r\n",
       "Q 37.359375 48.484375 30.078125 48.484375 \r\n",
       "Q 22.859375 48.484375 18.828125 43.109375 \r\n",
       "Q 14.796875 37.75 14.796875 27.984375 \r\n",
       "Q 14.796875 18.265625 18.828125 12.890625 \r\n",
       "Q 22.859375 7.515625 30.078125 7.515625 \r\n",
       "Q 37.359375 7.515625 41.375 12.890625 \r\n",
       "Q 45.40625 18.265625 45.40625 27.984375 \r\n",
       "z\r\n",
       "M 54.390625 6.78125 \r\n",
       "Q 54.390625 -7.171875 48.1875 -13.984375 \r\n",
       "Q 42 -20.796875 29.203125 -20.796875 \r\n",
       "Q 24.46875 -20.796875 20.265625 -20.09375 \r\n",
       "Q 16.0625 -19.390625 12.109375 -17.921875 \r\n",
       "L 12.109375 -9.1875 \r\n",
       "Q 16.0625 -11.328125 19.921875 -12.34375 \r\n",
       "Q 23.78125 -13.375 27.78125 -13.375 \r\n",
       "Q 36.625 -13.375 41.015625 -8.765625 \r\n",
       "Q 45.40625 -4.15625 45.40625 5.171875 \r\n",
       "L 45.40625 9.625 \r\n",
       "Q 42.625 4.78125 38.28125 2.390625 \r\n",
       "Q 33.9375 0 27.875 0 \r\n",
       "Q 17.828125 0 11.671875 7.65625 \r\n",
       "Q 5.515625 15.328125 5.515625 27.984375 \r\n",
       "Q 5.515625 40.671875 11.671875 48.328125 \r\n",
       "Q 17.828125 56 27.875 56 \r\n",
       "Q 33.9375 56 38.28125 53.609375 \r\n",
       "Q 42.625 51.21875 45.40625 46.390625 \r\n",
       "L 45.40625 54.6875 \r\n",
       "L 54.390625 54.6875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-67\"/>\r\n",
       "     <path d=\"M 48.6875 27.296875 \r\n",
       "Q 48.6875 37.203125 44.609375 42.84375 \r\n",
       "Q 40.53125 48.484375 33.40625 48.484375 \r\n",
       "Q 26.265625 48.484375 22.1875 42.84375 \r\n",
       "Q 18.109375 37.203125 18.109375 27.296875 \r\n",
       "Q 18.109375 17.390625 22.1875 11.75 \r\n",
       "Q 26.265625 6.109375 33.40625 6.109375 \r\n",
       "Q 40.53125 6.109375 44.609375 11.75 \r\n",
       "Q 48.6875 17.390625 48.6875 27.296875 \r\n",
       "z\r\n",
       "M 18.109375 46.390625 \r\n",
       "Q 20.953125 51.265625 25.265625 53.625 \r\n",
       "Q 29.59375 56 35.59375 56 \r\n",
       "Q 45.5625 56 51.78125 48.09375 \r\n",
       "Q 58.015625 40.1875 58.015625 27.296875 \r\n",
       "Q 58.015625 14.40625 51.78125 6.484375 \r\n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \r\n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \r\n",
       "Q 20.953125 3.328125 18.109375 8.203125 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 75.984375 \r\n",
       "L 18.109375 75.984375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-62\"/>\r\n",
       "    </defs>\r\n",
       "    <g transform=\"translate(179.977523 60.481826)scale(0.1 -0.1)\">\r\n",
       "     <use xlink:href=\"#DejaVuSans-67\"/>\r\n",
       "     <use x=\"63.476562\" xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "     <use x=\"91.259766\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "     <use x=\"152.441406\" xlink:href=\"#DejaVuSans-62\"/>\r\n",
       "     <use x=\"215.917969\" xlink:href=\"#DejaVuSans-61\"/>\r\n",
       "     <use x=\"277.197266\" xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "     <use x=\"304.980469\" xlink:href=\"#DejaVuSans-20\"/>\r\n",
       "     <use x=\"336.767578\" xlink:href=\"#DejaVuSans-6d\"/>\r\n",
       "     <use x=\"434.179688\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "     <use x=\"461.962891\" xlink:href=\"#DejaVuSans-6e\"/>\r\n",
       "     <use x=\"525.341797\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "     <use x=\"553.125\" xlink:href=\"#DejaVuSans-6d\"/>\r\n",
       "     <use x=\"650.537109\" xlink:href=\"#DejaVuSans-75\"/>\r\n",
       "     <use x=\"713.916016\" xlink:href=\"#DejaVuSans-6d\"/>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"pf01846f6f1\">\r\n",
       "   <rect height=\"135.9\" width=\"251.1\" x=\"42.620313\" y=\"10.7\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 324x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def f(x):\n",
    "    return x * np.cos(np.pi * x)\n",
    "\n",
    "dl_utils.set_figsize((4.5, 2.5))\n",
    "x = np.arange(-1.0, 2.0, 0.1)\n",
    "fig,  = dl_utils.plt.plot(x, f(x))\n",
    "fig.axes.annotate('local minimum', xy=(-0.3, -0.25), xytext=(-0.77, -1.0),\n",
    "                  arrowprops=dict(arrowstyle='->'))\n",
    "fig.axes.annotate('global minimum', xy=(1.1, -0.95), xytext=(0.6, 0.8),\n",
    "                  arrowprops=dict(arrowstyle='->'))\n",
    "dl_utils.plt.xlabel('x')\n",
    "dl_utils.plt.ylabel('f(x)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "深度学习模型的目标函数可能有若干局部最优值。当一个优化问题的数值解在局部最优解附近时，由于目标函数有关解的梯度接近或变成零，最终迭代求得的数值解可能只令目标函数局部最小化而非全局最小化。\n",
    "\n",
    "###  鞍点\n",
    "\n",
    "刚刚我们提到，**梯度接近或变成零**可能是由于当前解在局部最优解附近造成的。事实上，另一种可能性是当前解在鞍点（saddle point）附近。\n",
    "\n",
    "举个例子，给定函数\n",
    "\n",
    "$$f(x) = x^3,$$\n",
    "\n",
    "我们可以找出该函数的鞍点位置。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\r\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 304.420313 184.15625\" width=\"304.420313pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 184.15625 \r\n",
       "L 304.420313 184.15625 \r\n",
       "L 304.420313 -0 \r\n",
       "L 0 -0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 42.620313 146.6 \r\n",
       "L 293.720313 146.6 \r\n",
       "L 293.720313 10.7 \r\n",
       "L 42.620313 10.7 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"meafb72d447\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"54.033949\" xlink:href=\"#meafb72d447\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- −2 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\"/>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(46.662855 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"112.565417\" xlink:href=\"#meafb72d447\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- −1 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(105.194324 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"171.096886\" xlink:href=\"#meafb72d447\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(167.915636 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"229.628354\" xlink:href=\"#meafb72d447\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 1 -->\r\n",
       "      <g transform=\"translate(226.447104 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"288.159823\" xlink:href=\"#meafb72d447\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 2 -->\r\n",
       "      <g transform=\"translate(284.978573 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_6\">\r\n",
       "     <!-- x -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 54.890625 54.6875 \r\n",
       "L 35.109375 28.078125 \r\n",
       "L 55.90625 0 \r\n",
       "L 45.3125 0 \r\n",
       "L 29.390625 21.484375 \r\n",
       "L 13.484375 0 \r\n",
       "L 2.875 0 \r\n",
       "L 24.125 28.609375 \r\n",
       "L 4.6875 54.6875 \r\n",
       "L 15.28125 54.6875 \r\n",
       "L 29.78125 35.203125 \r\n",
       "L 44.28125 54.6875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(165.210938 174.876562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m9a2c407a22\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620313\" xlink:href=\"#m9a2c407a22\" y=\"115.479167\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- −5 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.796875 72.90625 \r\n",
       "L 49.515625 72.90625 \r\n",
       "L 49.515625 64.59375 \r\n",
       "L 19.828125 64.59375 \r\n",
       "L 19.828125 46.734375 \r\n",
       "Q 21.96875 47.46875 24.109375 47.828125 \r\n",
       "Q 26.265625 48.1875 28.421875 48.1875 \r\n",
       "Q 40.625 48.1875 47.75 41.5 \r\n",
       "Q 54.890625 34.8125 54.890625 23.390625 \r\n",
       "Q 54.890625 11.625 47.5625 5.09375 \r\n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \r\n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \r\n",
       "Q 12.796875 0.140625 7.71875 1.703125 \r\n",
       "L 7.71875 11.625 \r\n",
       "Q 12.109375 9.234375 16.796875 8.0625 \r\n",
       "Q 21.484375 6.890625 26.703125 6.890625 \r\n",
       "Q 35.15625 6.890625 40.078125 11.328125 \r\n",
       "Q 45.015625 15.765625 45.015625 23.390625 \r\n",
       "Q 45.015625 31 40.078125 35.4375 \r\n",
       "Q 35.15625 39.890625 26.703125 39.890625 \r\n",
       "Q 22.75 39.890625 18.8125 39.015625 \r\n",
       "Q 14.890625 38.140625 10.796875 36.28125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-35\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 119.278386)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620313\" xlink:href=\"#m9a2c407a22\" y=\"73.906566\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(29.257813 77.705785)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620313\" xlink:href=\"#m9a2c407a22\" y=\"32.333966\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 5 -->\r\n",
       "      <g transform=\"translate(29.257813 36.133184)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_10\">\r\n",
       "     <!-- f(x) -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 37.109375 75.984375 \r\n",
       "L 37.109375 68.5 \r\n",
       "L 28.515625 68.5 \r\n",
       "Q 23.6875 68.5 21.796875 66.546875 \r\n",
       "Q 19.921875 64.59375 19.921875 59.515625 \r\n",
       "L 19.921875 54.6875 \r\n",
       "L 34.71875 54.6875 \r\n",
       "L 34.71875 47.703125 \r\n",
       "L 19.921875 47.703125 \r\n",
       "L 19.921875 0 \r\n",
       "L 10.890625 0 \r\n",
       "L 10.890625 47.703125 \r\n",
       "L 2.296875 47.703125 \r\n",
       "L 2.296875 54.6875 \r\n",
       "L 10.890625 54.6875 \r\n",
       "L 10.890625 58.5 \r\n",
       "Q 10.890625 67.625 15.140625 71.796875 \r\n",
       "Q 19.390625 75.984375 28.609375 75.984375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-66\"/>\r\n",
       "      <path d=\"M 31 75.875 \r\n",
       "Q 24.46875 64.65625 21.28125 53.65625 \r\n",
       "Q 18.109375 42.671875 18.109375 31.390625 \r\n",
       "Q 18.109375 20.125 21.3125 9.0625 \r\n",
       "Q 24.515625 -2 31 -13.1875 \r\n",
       "L 23.1875 -13.1875 \r\n",
       "Q 15.875 -1.703125 12.234375 9.375 \r\n",
       "Q 8.59375 20.453125 8.59375 31.390625 \r\n",
       "Q 8.59375 42.28125 12.203125 53.3125 \r\n",
       "Q 15.828125 64.359375 23.1875 75.875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-28\"/>\r\n",
       "      <path d=\"M 8.015625 75.875 \r\n",
       "L 15.828125 75.875 \r\n",
       "Q 23.140625 64.359375 26.78125 53.3125 \r\n",
       "Q 30.421875 42.28125 30.421875 31.390625 \r\n",
       "Q 30.421875 20.453125 26.78125 9.375 \r\n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \r\n",
       "L 8.015625 -13.1875 \r\n",
       "Q 14.5 -2 17.703125 9.0625 \r\n",
       "Q 20.90625 20.125 20.90625 31.390625 \r\n",
       "Q 20.90625 42.671875 17.703125 53.65625 \r\n",
       "Q 14.5 64.65625 8.015625 75.875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-29\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798438 87.271094)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\r\n",
       "      <use x=\"35.205078\" xlink:href=\"#DejaVuSans-28\"/>\r\n",
       "      <use x=\"74.21875\" xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "      <use x=\"133.398438\" xlink:href=\"#DejaVuSans-29\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_9\">\r\n",
       "    <path clip-path=\"url(#p97589d4e29)\" d=\"M 54.033949 140.422727 \r\n",
       "L 59.887096 130.93586 \r\n",
       "L 65.740243 122.396848 \r\n",
       "L 71.593389 114.755804 \r\n",
       "L 77.446536 107.962841 \r\n",
       "L 83.299683 101.968072 \r\n",
       "L 89.15283 96.721609 \r\n",
       "L 95.005977 92.173567 \r\n",
       "L 100.859124 88.274057 \r\n",
       "L 106.712271 84.973193 \r\n",
       "L 112.565417 82.221086 \r\n",
       "L 118.418564 79.967851 \r\n",
       "L 124.271711 78.163601 \r\n",
       "L 130.124858 76.758447 \r\n",
       "L 135.978005 75.702503 \r\n",
       "L 141.831152 74.945881 \r\n",
       "L 147.684299 74.438696 \r\n",
       "L 153.537445 74.131058 \r\n",
       "L 159.390592 73.973082 \r\n",
       "L 165.243739 73.914881 \r\n",
       "L 171.096886 73.906566 \r\n",
       "L 176.950033 73.898252 \r\n",
       "L 182.80318 73.84005 \r\n",
       "L 188.656326 73.682074 \r\n",
       "L 194.509473 73.374437 \r\n",
       "L 200.36262 72.867251 \r\n",
       "L 206.215767 72.11063 \r\n",
       "L 212.068914 71.054686 \r\n",
       "L 217.922061 69.649532 \r\n",
       "L 223.775208 67.845281 \r\n",
       "L 229.628354 65.592046 \r\n",
       "L 235.481501 62.83994 \r\n",
       "L 241.334648 59.539075 \r\n",
       "L 247.187795 55.639566 \r\n",
       "L 253.040942 51.091523 \r\n",
       "L 258.894089 45.845061 \r\n",
       "L 264.747236 39.850292 \r\n",
       "L 270.600382 33.057329 \r\n",
       "L 276.453529 25.416285 \r\n",
       "L 282.306676 16.877273 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 42.620313 146.6 \r\n",
       "L 42.620313 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 293.720313 146.6 \r\n",
       "L 293.720313 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 42.620313 146.6 \r\n",
       "L 293.720313 146.6 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 42.620313 10.7 \r\n",
       "L 293.720313 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_7\">\r\n",
       "    <path d=\"M 171.389768 103.881861 \r\n",
       "Q 171.253999 91.142742 171.130145 79.521593 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\r\n",
       "    <path d=\"M 169.172887 83.54268 \r\n",
       "L 171.130145 79.521593 \r\n",
       "L 173.17266 83.500052 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"text_11\">\r\n",
       "    <!-- saddle point -->\r\n",
       "    <defs>\r\n",
       "     <path d=\"M 44.28125 53.078125 \r\n",
       "L 44.28125 44.578125 \r\n",
       "Q 40.484375 46.53125 36.375 47.5 \r\n",
       "Q 32.28125 48.484375 27.875 48.484375 \r\n",
       "Q 21.1875 48.484375 17.84375 46.4375 \r\n",
       "Q 14.5 44.390625 14.5 40.28125 \r\n",
       "Q 14.5 37.15625 16.890625 35.375 \r\n",
       "Q 19.28125 33.59375 26.515625 31.984375 \r\n",
       "L 29.59375 31.296875 \r\n",
       "Q 39.15625 29.25 43.1875 25.515625 \r\n",
       "Q 47.21875 21.78125 47.21875 15.09375 \r\n",
       "Q 47.21875 7.46875 41.1875 3.015625 \r\n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \r\n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \r\n",
       "Q 10.6875 0.296875 5.421875 2 \r\n",
       "L 5.421875 11.28125 \r\n",
       "Q 10.40625 8.6875 15.234375 7.390625 \r\n",
       "Q 20.0625 6.109375 24.8125 6.109375 \r\n",
       "Q 31.15625 6.109375 34.5625 8.28125 \r\n",
       "Q 37.984375 10.453125 37.984375 14.40625 \r\n",
       "Q 37.984375 18.0625 35.515625 20.015625 \r\n",
       "Q 33.0625 21.96875 24.703125 23.78125 \r\n",
       "L 21.578125 24.515625 \r\n",
       "Q 13.234375 26.265625 9.515625 29.90625 \r\n",
       "Q 5.8125 33.546875 5.8125 39.890625 \r\n",
       "Q 5.8125 47.609375 11.28125 51.796875 \r\n",
       "Q 16.75 56 26.8125 56 \r\n",
       "Q 31.78125 56 36.171875 55.265625 \r\n",
       "Q 40.578125 54.546875 44.28125 53.078125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-73\"/>\r\n",
       "     <path d=\"M 34.28125 27.484375 \r\n",
       "Q 23.390625 27.484375 19.1875 25 \r\n",
       "Q 14.984375 22.515625 14.984375 16.5 \r\n",
       "Q 14.984375 11.71875 18.140625 8.90625 \r\n",
       "Q 21.296875 6.109375 26.703125 6.109375 \r\n",
       "Q 34.1875 6.109375 38.703125 11.40625 \r\n",
       "Q 43.21875 16.703125 43.21875 25.484375 \r\n",
       "L 43.21875 27.484375 \r\n",
       "z\r\n",
       "M 52.203125 31.203125 \r\n",
       "L 52.203125 0 \r\n",
       "L 43.21875 0 \r\n",
       "L 43.21875 8.296875 \r\n",
       "Q 40.140625 3.328125 35.546875 0.953125 \r\n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \r\n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \r\n",
       "Q 6 8.015625 6 15.921875 \r\n",
       "Q 6 25.140625 12.171875 29.828125 \r\n",
       "Q 18.359375 34.515625 30.609375 34.515625 \r\n",
       "L 43.21875 34.515625 \r\n",
       "L 43.21875 35.40625 \r\n",
       "Q 43.21875 41.609375 39.140625 45 \r\n",
       "Q 35.0625 48.390625 27.6875 48.390625 \r\n",
       "Q 23 48.390625 18.546875 47.265625 \r\n",
       "Q 14.109375 46.140625 10.015625 43.890625 \r\n",
       "L 10.015625 52.203125 \r\n",
       "Q 14.9375 54.109375 19.578125 55.046875 \r\n",
       "Q 24.21875 56 28.609375 56 \r\n",
       "Q 40.484375 56 46.34375 49.84375 \r\n",
       "Q 52.203125 43.703125 52.203125 31.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-61\"/>\r\n",
       "     <path d=\"M 45.40625 46.390625 \r\n",
       "L 45.40625 75.984375 \r\n",
       "L 54.390625 75.984375 \r\n",
       "L 54.390625 0 \r\n",
       "L 45.40625 0 \r\n",
       "L 45.40625 8.203125 \r\n",
       "Q 42.578125 3.328125 38.25 0.953125 \r\n",
       "Q 33.9375 -1.421875 27.875 -1.421875 \r\n",
       "Q 17.96875 -1.421875 11.734375 6.484375 \r\n",
       "Q 5.515625 14.40625 5.515625 27.296875 \r\n",
       "Q 5.515625 40.1875 11.734375 48.09375 \r\n",
       "Q 17.96875 56 27.875 56 \r\n",
       "Q 33.9375 56 38.25 53.625 \r\n",
       "Q 42.578125 51.265625 45.40625 46.390625 \r\n",
       "z\r\n",
       "M 14.796875 27.296875 \r\n",
       "Q 14.796875 17.390625 18.875 11.75 \r\n",
       "Q 22.953125 6.109375 30.078125 6.109375 \r\n",
       "Q 37.203125 6.109375 41.296875 11.75 \r\n",
       "Q 45.40625 17.390625 45.40625 27.296875 \r\n",
       "Q 45.40625 37.203125 41.296875 42.84375 \r\n",
       "Q 37.203125 48.484375 30.078125 48.484375 \r\n",
       "Q 22.953125 48.484375 18.875 42.84375 \r\n",
       "Q 14.796875 37.203125 14.796875 27.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-64\"/>\r\n",
       "     <path d=\"M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6c\"/>\r\n",
       "     <path d=\"M 56.203125 29.59375 \r\n",
       "L 56.203125 25.203125 \r\n",
       "L 14.890625 25.203125 \r\n",
       "Q 15.484375 15.921875 20.484375 11.0625 \r\n",
       "Q 25.484375 6.203125 34.421875 6.203125 \r\n",
       "Q 39.59375 6.203125 44.453125 7.46875 \r\n",
       "Q 49.3125 8.734375 54.109375 11.28125 \r\n",
       "L 54.109375 2.78125 \r\n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \r\n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \r\n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \r\n",
       "Q 5.515625 13.8125 5.515625 26.8125 \r\n",
       "Q 5.515625 40.234375 12.765625 48.109375 \r\n",
       "Q 20.015625 56 32.328125 56 \r\n",
       "Q 43.359375 56 49.78125 48.890625 \r\n",
       "Q 56.203125 41.796875 56.203125 29.59375 \r\n",
       "z\r\n",
       "M 47.21875 32.234375 \r\n",
       "Q 47.125 39.59375 43.09375 43.984375 \r\n",
       "Q 39.0625 48.390625 32.421875 48.390625 \r\n",
       "Q 24.90625 48.390625 20.390625 44.140625 \r\n",
       "Q 15.875 39.890625 15.1875 32.171875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-65\"/>\r\n",
       "     <path id=\"DejaVuSans-20\"/>\r\n",
       "     <path d=\"M 18.109375 8.203125 \r\n",
       "L 18.109375 -20.796875 \r\n",
       "L 9.078125 -20.796875 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.390625 \r\n",
       "Q 20.953125 51.265625 25.265625 53.625 \r\n",
       "Q 29.59375 56 35.59375 56 \r\n",
       "Q 45.5625 56 51.78125 48.09375 \r\n",
       "Q 58.015625 40.1875 58.015625 27.296875 \r\n",
       "Q 58.015625 14.40625 51.78125 6.484375 \r\n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \r\n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \r\n",
       "Q 20.953125 3.328125 18.109375 8.203125 \r\n",
       "z\r\n",
       "M 48.6875 27.296875 \r\n",
       "Q 48.6875 37.203125 44.609375 42.84375 \r\n",
       "Q 40.53125 48.484375 33.40625 48.484375 \r\n",
       "Q 26.265625 48.484375 22.1875 42.84375 \r\n",
       "Q 18.109375 37.203125 18.109375 27.296875 \r\n",
       "Q 18.109375 17.390625 22.1875 11.75 \r\n",
       "Q 26.265625 6.109375 33.40625 6.109375 \r\n",
       "Q 40.53125 6.109375 44.609375 11.75 \r\n",
       "Q 48.6875 17.390625 48.6875 27.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-70\"/>\r\n",
       "     <path d=\"M 30.609375 48.390625 \r\n",
       "Q 23.390625 48.390625 19.1875 42.75 \r\n",
       "Q 14.984375 37.109375 14.984375 27.296875 \r\n",
       "Q 14.984375 17.484375 19.15625 11.84375 \r\n",
       "Q 23.34375 6.203125 30.609375 6.203125 \r\n",
       "Q 37.796875 6.203125 41.984375 11.859375 \r\n",
       "Q 46.1875 17.53125 46.1875 27.296875 \r\n",
       "Q 46.1875 37.015625 41.984375 42.703125 \r\n",
       "Q 37.796875 48.390625 30.609375 48.390625 \r\n",
       "z\r\n",
       "M 30.609375 56 \r\n",
       "Q 42.328125 56 49.015625 48.375 \r\n",
       "Q 55.71875 40.765625 55.71875 27.296875 \r\n",
       "Q 55.71875 13.875 49.015625 6.21875 \r\n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \r\n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \r\n",
       "Q 5.515625 13.875 5.515625 27.296875 \r\n",
       "Q 5.515625 40.765625 12.171875 48.375 \r\n",
       "Q 18.84375 56 30.609375 56 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6f\"/>\r\n",
       "     <path d=\"M 9.421875 54.6875 \r\n",
       "L 18.40625 54.6875 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 64.59375 \r\n",
       "L 9.421875 64.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-69\"/>\r\n",
       "     <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6e\"/>\r\n",
       "     <path d=\"M 18.3125 70.21875 \r\n",
       "L 18.3125 54.6875 \r\n",
       "L 36.8125 54.6875 \r\n",
       "L 36.8125 47.703125 \r\n",
       "L 18.3125 47.703125 \r\n",
       "L 18.3125 18.015625 \r\n",
       "Q 18.3125 11.328125 20.140625 9.421875 \r\n",
       "Q 21.96875 7.515625 27.59375 7.515625 \r\n",
       "L 36.8125 7.515625 \r\n",
       "L 36.8125 0 \r\n",
       "L 27.59375 0 \r\n",
       "Q 17.1875 0 13.234375 3.875 \r\n",
       "Q 9.28125 7.765625 9.28125 18.015625 \r\n",
       "L 9.28125 47.703125 \r\n",
       "L 2.6875 47.703125 \r\n",
       "L 2.6875 54.6875 \r\n",
       "L 9.28125 54.6875 \r\n",
       "L 9.28125 70.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-74\"/>\r\n",
       "    </defs>\r\n",
       "    <g transform=\"translate(140.660522 115.479167)scale(0.1 -0.1)\">\r\n",
       "     <use xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "     <use x=\"52.099609\" xlink:href=\"#DejaVuSans-61\"/>\r\n",
       "     <use x=\"113.378906\" xlink:href=\"#DejaVuSans-64\"/>\r\n",
       "     <use x=\"176.855469\" xlink:href=\"#DejaVuSans-64\"/>\r\n",
       "     <use x=\"240.332031\" xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "     <use x=\"268.115234\" xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "     <use x=\"329.638672\" xlink:href=\"#DejaVuSans-20\"/>\r\n",
       "     <use x=\"361.425781\" xlink:href=\"#DejaVuSans-70\"/>\r\n",
       "     <use x=\"424.902344\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "     <use x=\"486.083984\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "     <use x=\"513.867188\" xlink:href=\"#DejaVuSans-6e\"/>\r\n",
       "     <use x=\"577.246094\" xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p97589d4e29\">\r\n",
       "   <rect height=\"135.9\" width=\"251.1\" x=\"42.620313\" y=\"10.7\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 324x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(-2.0, 2.0, 0.1)\n",
    "fig, = dl_utils.plt.plot(x, x**3)\n",
    "fig.axes.annotate('saddle point', xy=(0, -0.3), xytext=(-0.52, -5.0),\n",
    "                  arrowprops=dict(arrowstyle='->'))\n",
    "dl_utils.plt.xlabel('x')\n",
    "dl_utils.plt.ylabel('f(x)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再举个定义在二维空间的函数的例子，例如：\n",
    "\n",
    "$$f(x, y) = x^2 - y^2.$$\n",
    "\n",
    "我们可以找出该函数的鞍点位置。也许你已经发现了，该函数看起来像一个马鞍，而鞍点恰好是马鞍上可坐区域的中心。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\r\n",
       "<svg height=\"157.3pt\" version=\"1.1\" viewBox=\"0 0 272.5 157.3\" width=\"272.5pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 157.3 \r\n",
       "L 272.5 157.3 \r\n",
       "L 272.5 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"patch_2\">\r\n",
       "   <path d=\"M 10.7 146.6 \r\n",
       "L 261.8 146.6 \r\n",
       "L 261.8 10.7 \r\n",
       "L 10.7 10.7 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"pane3d_1\">\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 43.987185 116.570889 \r\n",
       "L 115.853319 83.251289 \r\n",
       "L 114.656824 19.583 \r\n",
       "L 38.783682 49.519976 \r\n",
       "\" style=\"fill:#f2f2f2;opacity:0.5;stroke:#f2f2f2;stroke-linejoin:miter;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       "  <g id=\"pane3d_2\">\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 115.853319 83.251289 \r\n",
       "L 232.269699 101.879096 \r\n",
       "L 237.140421 36.290916 \r\n",
       "L 114.656824 19.583 \r\n",
       "\" style=\"fill:#e6e6e6;opacity:0.5;stroke:#e6e6e6;stroke-linejoin:miter;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       "  <g id=\"pane3d_3\">\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 43.987185 116.570889 \r\n",
       "L 166.472947 138.266006 \r\n",
       "L 232.269699 101.879096 \r\n",
       "L 115.853319 83.251289 \r\n",
       "\" style=\"fill:#ececec;opacity:0.5;stroke:#ececec;stroke-linejoin:miter;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       "  <g id=\"axis3d_1\">\r\n",
       "   <g id=\"line2d_1\">\r\n",
       "    <path d=\"M 43.987185 116.570889 \r\n",
       "L 166.472947 138.266006 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"text_1\">\r\n",
       "    <!-- x -->\r\n",
       "    <defs>\r\n",
       "     <path d=\"M 54.890625 54.6875 \r\n",
       "L 35.109375 28.078125 \r\n",
       "L 55.90625 0 \r\n",
       "L 45.3125 0 \r\n",
       "L 29.390625 21.484375 \r\n",
       "L 13.484375 0 \r\n",
       "L 2.875 0 \r\n",
       "L 24.125 28.609375 \r\n",
       "L 4.6875 54.6875 \r\n",
       "L 15.28125 54.6875 \r\n",
       "L 29.78125 35.203125 \r\n",
       "L 44.28125 54.6875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\"/>\r\n",
       "    </defs>\r\n",
       "    <g transform=\"translate(82.750785 155.409667)scale(0.1 -0.1)\">\r\n",
       "     <use xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"Line3DCollection_1\">\r\n",
       "    <path d=\"M 51.454708 117.893564 \r\n",
       "L 122.973198 84.390542 \r\n",
       "L 122.132462 20.602748 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\r\n",
       "    <path d=\"M 103.83065 127.170578 \r\n",
       "L 172.828996 92.367978 \r\n",
       "L 174.535205 27.750975 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\r\n",
       "    <path d=\"M 158.338065 136.825126 \r\n",
       "L 224.562087 100.645799 \r\n",
       "L 229.014604 35.182478 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"xtick_1\">\r\n",
       "    <g id=\"line2d_2\">\r\n",
       "     <path d=\"M 52.070877 117.604918 \r\n",
       "L 50.220073 118.471932 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_2\">\r\n",
       "     <!-- −1 -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\"/>\r\n",
       "      <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(35.785551 136.914626)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "      <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"xtick_2\">\r\n",
       "    <g id=\"line2d_3\">\r\n",
       "     <path d=\"M 104.425952 126.870309 \r\n",
       "L 102.63778 127.772259 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_3\">\r\n",
       "     <!-- 0 -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(92.343524 146.448134)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"xtick_3\">\r\n",
       "    <g id=\"line2d_4\">\r\n",
       "     <path d=\"M 158.910277 136.512517 \r\n",
       "L 157.19142 137.451558 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_4\">\r\n",
       "     <!-- 1 -->\r\n",
       "     <g transform=\"translate(146.849377 156.37081)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       "  <g id=\"axis3d_2\">\r\n",
       "   <g id=\"line2d_5\">\r\n",
       "    <path d=\"M 232.269699 101.879096 \r\n",
       "L 166.472947 138.266006 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"text_5\">\r\n",
       "    <!-- y -->\r\n",
       "    <defs>\r\n",
       "     <path d=\"M 32.171875 -5.078125 \r\n",
       "Q 28.375 -14.84375 24.75 -17.8125 \r\n",
       "Q 21.140625 -20.796875 15.09375 -20.796875 \r\n",
       "L 7.90625 -20.796875 \r\n",
       "L 7.90625 -13.28125 \r\n",
       "L 13.1875 -13.28125 \r\n",
       "Q 16.890625 -13.28125 18.9375 -11.515625 \r\n",
       "Q 21 -9.765625 23.484375 -3.21875 \r\n",
       "L 25.09375 0.875 \r\n",
       "L 2.984375 54.6875 \r\n",
       "L 12.5 54.6875 \r\n",
       "L 29.59375 11.921875 \r\n",
       "L 46.6875 54.6875 \r\n",
       "L 56.203125 54.6875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-79\"/>\r\n",
       "    </defs>\r\n",
       "    <g transform=\"translate(227.803284 143.124581)scale(0.1 -0.1)\">\r\n",
       "     <use xlink:href=\"#DejaVuSans-79\"/>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"Line3DCollection_2\">\r\n",
       "    <path d=\"M 43.978971 47.470091 \r\n",
       "L 48.890254 114.297659 \r\n",
       "L 170.975981 135.775739 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\r\n",
       "    <path d=\"M 78.207339 33.964735 \r\n",
       "L 81.25884 99.290474 \r\n",
       "L 200.651948 119.36434 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\r\n",
       "    <path d=\"M 110.171251 21.352856 \r\n",
       "L 111.589022 85.228363 \r\n",
       "L 228.377837 104.031372 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"xtick_4\">\r\n",
       "    <g id=\"line2d_6\">\r\n",
       "     <path d=\"M 169.954231 135.595986 \r\n",
       "L 173.021765 136.135646 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_6\">\r\n",
       "     <!-- −1 -->\r\n",
       "     <g transform=\"translate(177.98021 153.369862)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "      <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"xtick_5\">\r\n",
       "    <g id=\"line2d_7\">\r\n",
       "     <path d=\"M 199.654229 119.196591 \r\n",
       "L 202.649539 119.7002 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_7\">\r\n",
       "     <!-- 0 -->\r\n",
       "     <g transform=\"translate(211.334516 136.632167)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"xtick_6\">\r\n",
       "    <g id=\"line2d_8\">\r\n",
       "     <path d=\"M 227.40325 103.874464 \r\n",
       "L 230.329048 104.345517 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_8\">\r\n",
       "     <!-- 1 -->\r\n",
       "     <g transform=\"translate(238.583983 120.993501)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       "  <g id=\"axis3d_3\">\r\n",
       "   <g id=\"line2d_9\">\r\n",
       "    <path d=\"M 232.269699 101.879096 \r\n",
       "L 237.140421 36.290916 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"Line3DCollection_3\">\r\n",
       "    <path d=\"M 232.362339 100.631618 \r\n",
       "L 115.830513 82.037737 \r\n",
       "L 43.888382 115.297743 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\r\n",
       "    <path d=\"M 234.642669 69.925148 \r\n",
       "L 115.269747 52.198043 \r\n",
       "L 41.454325 83.933148 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\r\n",
       "    <path d=\"M 237.037992 37.670208 \r\n",
       "L 114.681932 20.919065 \r\n",
       "L 38.893294 50.932407 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"xtick_7\">\r\n",
       "    <g id=\"line2d_10\">\r\n",
       "     <path d=\"M 231.390048 100.476479 \r\n",
       "L 234.308945 100.94222 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_9\">\r\n",
       "     <!-- −1 -->\r\n",
       "     <g transform=\"translate(245.740836 106.615424)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "      <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"xtick_8\">\r\n",
       "    <g id=\"line2d_11\">\r\n",
       "     <path d=\"M 233.645664 69.777091 \r\n",
       "L 236.638804 70.221578 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_10\">\r\n",
       "     <!-- 0 -->\r\n",
       "     <g transform=\"translate(252.711018 75.982787)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"xtick_9\">\r\n",
       "    <g id=\"line2d_12\">\r\n",
       "     <path d=\"M 236.014986 37.530153 \r\n",
       "L 239.086238 37.950623 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_11\">\r\n",
       "     <!-- 1 -->\r\n",
       "     <g transform=\"translate(255.631033 43.811251)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"Line3DCollection_4\">\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 54.133789 83.013765 \r\n",
       "L 56.139919 77.877046 \r\n",
       "L 58.168328 73.018701 \r\n",
       "L 60.21769 68.442641 \r\n",
       "L 62.286647 64.152404 \r\n",
       "L 64.373813 60.151146 \r\n",
       "L 66.477777 56.441634 \r\n",
       "L 68.597104 53.026238 \r\n",
       "L 70.730341 49.906927 \r\n",
       "L 72.876017 47.085267 \r\n",
       "L 75.032652 44.562414 \r\n",
       "L 77.198755 42.339118 \r\n",
       "L 79.372827 40.415722 \r\n",
       "L 81.55337 38.792162 \r\n",
       "L 83.738887 37.467973 \r\n",
       "L 85.927882 36.44229 \r\n",
       "L 88.118869 35.713859 \r\n",
       "L 90.310374 35.281039 \r\n",
       "L 92.500935 35.141811 \r\n",
       "L 94.689107 35.293791 \r\n",
       "L 96.873466 35.734237 \r\n",
       "L 99.052611 36.460061 \r\n",
       "L 101.225166 37.467842 \r\n",
       "L 103.389782 38.753839 \r\n",
       "L 105.545142 40.314003 \r\n",
       "L 107.689962 42.143996 \r\n",
       "L 109.822991 44.239201 \r\n",
       "L 111.943017 46.594743 \r\n",
       "L 114.048865 49.205501 \r\n",
       "L 116.1394 52.066124 \r\n",
       "L 118.213531 55.171053 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 61.646625 92.119798 \r\n",
       "L 63.65267 87.010529 \r\n",
       "L 65.679222 82.176611 \r\n",
       "L 67.724956 77.621918 \r\n",
       "L 69.788521 73.349956 \r\n",
       "L 71.868537 69.363853 \r\n",
       "L 73.963603 65.666357 \r\n",
       "L 76.072295 62.259822 \r\n",
       "L 78.193176 59.146208 \r\n",
       "L 80.324791 56.327075 \r\n",
       "L 82.46568 53.803584 \r\n",
       "L 84.614371 51.576492 \r\n",
       "L 86.769392 49.646154 \r\n",
       "L 88.929268 48.012527 \r\n",
       "L 91.09253 46.675169 \r\n",
       "L 93.257713 45.633247 \r\n",
       "L 95.423361 44.885539 \r\n",
       "L 97.588033 44.430447 \r\n",
       "L 99.750301 44.265997 \r\n",
       "L 101.908758 44.389854 \r\n",
       "L 104.062017 44.799331 \r\n",
       "L 106.208714 45.491399 \r\n",
       "L 108.347515 46.462699 \r\n",
       "L 110.47711 47.709557 \r\n",
       "L 112.596226 49.227995 \r\n",
       "L 114.70362 51.013747 \r\n",
       "L 116.798086 53.062274 \r\n",
       "L 118.878454 55.368778 \r\n",
       "L 120.943595 57.928221 \r\n",
       "L 122.992419 60.735335 \r\n",
       "L 125.023878 63.784647 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 69.041795 100.041427 \r\n",
       "L 71.049442 94.953716 \r\n",
       "L 73.075882 90.138857 \r\n",
       "L 75.119791 85.600694 \r\n",
       "L 77.17982 81.342708 \r\n",
       "L 79.254594 77.36801 \r\n",
       "L 81.342717 73.679329 \r\n",
       "L 83.442778 70.279009 \r\n",
       "L 85.553348 67.169004 \r\n",
       "L 87.672989 64.350872 \r\n",
       "L 89.800255 61.825776 \r\n",
       "L 91.933694 59.59448 \r\n",
       "L 94.071854 57.657352 \r\n",
       "L 96.213282 56.014365 \r\n",
       "L 98.356534 54.665097 \r\n",
       "L 100.50017 53.60874 \r\n",
       "L 102.642764 52.844103 \r\n",
       "L 104.782905 52.369619 \r\n",
       "L 106.919195 52.183353 \r\n",
       "L 109.050261 52.283009 \r\n",
       "L 111.17475 52.665947 \r\n",
       "L 113.291334 53.329183 \r\n",
       "L 115.398714 54.269412 \r\n",
       "L 117.495623 55.483013 \r\n",
       "L 119.580822 56.966064 \r\n",
       "L 121.653111 58.714361 \r\n",
       "L 123.711322 60.723425 \r\n",
       "L 125.75433 62.988523 \r\n",
       "L 127.781044 65.504683 \r\n",
       "L 129.790419 68.266705 \r\n",
       "L 131.781449 71.269186 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 76.340106 106.809598 \r\n",
       "L 78.351052 101.737687 \r\n",
       "L 80.379127 96.936635 \r\n",
       "L 82.423004 92.410265 \r\n",
       "L 84.481334 88.162041 \r\n",
       "L 86.552745 84.195059 \r\n",
       "L 88.635845 80.512037 \r\n",
       "L 90.729228 77.115313 \r\n",
       "L 92.831477 74.006837 \r\n",
       "L 94.941164 71.188167 \r\n",
       "L 97.056856 68.660469 \r\n",
       "L 99.177117 66.424515 \r\n",
       "L 101.300512 64.480683 \r\n",
       "L 103.425609 62.828958 \r\n",
       "L 105.550983 61.468936 \r\n",
       "L 107.675221 60.399831 \r\n",
       "L 109.79692 59.620473 \r\n",
       "L 111.914695 59.129323 \r\n",
       "L 114.027179 58.924474 \r\n",
       "L 116.133028 59.003666 \r\n",
       "L 118.230921 59.36429 \r\n",
       "L 120.319563 60.003402 \r\n",
       "L 122.39769 60.917736 \r\n",
       "L 124.464068 62.103713 \r\n",
       "L 126.517497 63.557458 \r\n",
       "L 128.556813 65.274811 \r\n",
       "L 130.580887 67.251343 \r\n",
       "L 132.588633 69.48237 \r\n",
       "L 134.579001 71.962971 \r\n",
       "L 136.550983 74.688 \r\n",
       "L 138.503618 77.652106 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 83.561656 112.448787 \r\n",
       "L 85.577618 107.387018 \r\n",
       "L 87.609084 102.594606 \r\n",
       "L 89.654723 98.075362 \r\n",
       "L 91.713182 93.83274 \r\n",
       "L 93.78309 89.869828 \r\n",
       "L 95.863058 86.189338 \r\n",
       "L 97.951683 82.793606 \r\n",
       "L 100.047554 79.684579 \r\n",
       "L 102.149252 76.863819 \r\n",
       "L 104.255356 74.332495 \r\n",
       "L 106.36444 72.091386 \r\n",
       "L 108.475087 70.140878 \r\n",
       "L 110.585879 68.480969 \r\n",
       "L 112.695413 67.111268 \r\n",
       "L 114.802294 66.031003 \r\n",
       "L 116.905143 65.239025 \r\n",
       "L 119.002599 64.733812 \r\n",
       "L 121.093321 64.513482 \r\n",
       "L 123.175992 64.575797 \r\n",
       "L 125.24932 64.918175 \r\n",
       "L 127.312043 65.537701 \r\n",
       "L 129.362926 66.431136 \r\n",
       "L 131.400769 67.594933 \r\n",
       "L 133.424407 69.02525 \r\n",
       "L 135.43271 70.71796 \r\n",
       "L 137.424588 72.668669 \r\n",
       "L 139.398988 74.872729 \r\n",
       "L 141.3549 77.325258 \r\n",
       "L 143.291356 80.021146 \r\n",
       "L 145.207431 82.955082 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 90.726022 116.977284 \r\n",
       "L 92.748743 111.920057 \r\n",
       "L 94.785375 107.13117 \r\n",
       "L 96.834578 102.614427 \r\n",
       "L 98.894996 98.37328 \r\n",
       "L 100.965254 94.410814 \r\n",
       "L 103.043963 90.729742 \r\n",
       "L 105.129723 87.332399 \r\n",
       "L 107.221125 84.220736 \r\n",
       "L 109.316757 81.396316 \r\n",
       "L 111.415205 78.860316 \r\n",
       "L 113.515055 76.613519 \r\n",
       "L 115.6149 74.65632 \r\n",
       "L 117.713339 72.988724 \r\n",
       "L 119.808983 71.610353 \r\n",
       "L 121.900456 70.520445 \r\n",
       "L 123.9864 69.717864 \r\n",
       "L 126.065474 69.201102 \r\n",
       "L 128.136362 68.968291 \r\n",
       "L 130.197772 69.017211 \r\n",
       "L 132.248438 69.345295 \r\n",
       "L 134.287126 69.949646 \r\n",
       "L 136.312633 70.827047 \r\n",
       "L 138.323788 71.973968 \r\n",
       "L 140.319459 73.386589 \r\n",
       "L 142.298549 75.060803 \r\n",
       "L 144.260003 76.99224 \r\n",
       "L 146.202803 79.176274 \r\n",
       "L 148.125975 81.608045 \r\n",
       "L 150.028589 84.282469 \r\n",
       "L 151.909757 87.194258 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 97.852434 120.407381 \r\n",
       "L 99.883695 115.349123 \r\n",
       "L 101.927294 110.558663 \r\n",
       "L 103.981884 106.039811 \r\n",
       "L 106.0461 101.796019 \r\n",
       "L 108.118564 97.830379 \r\n",
       "L 110.197881 94.145606 \r\n",
       "L 112.282653 90.744042 \r\n",
       "L 114.371471 87.627641 \r\n",
       "L 116.462927 84.797973 \r\n",
       "L 118.555611 82.25622 \r\n",
       "L 120.648119 80.003172 \r\n",
       "L 122.739052 78.039229 \r\n",
       "L 124.827022 76.364404 \r\n",
       "L 126.910653 74.978325 \r\n",
       "L 128.988586 73.880239 \r\n",
       "L 131.059478 73.069014 \r\n",
       "L 133.12201 72.543154 \r\n",
       "L 135.174887 72.300797 \r\n",
       "L 137.216839 72.33973 \r\n",
       "L 139.246627 72.657396 \r\n",
       "L 141.263041 73.250906 \r\n",
       "L 143.264906 74.117051 \r\n",
       "L 145.251081 75.252312 \r\n",
       "L 147.220464 76.652876 \r\n",
       "L 149.171991 78.314647 \r\n",
       "L 151.104637 80.233261 \r\n",
       "L 153.017421 82.404106 \r\n",
       "L 154.909403 84.822328 \r\n",
       "L 156.779688 87.482853 \r\n",
       "L 158.627425 90.380404 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 104.959949 122.745487 \r\n",
       "L 107.001576 117.680612 \r\n",
       "L 109.053982 112.883471 \r\n",
       "L 111.11581 108.357882 \r\n",
       "L 113.185685 104.107311 \r\n",
       "L 115.26222 100.134858 \r\n",
       "L 117.344017 96.443248 \r\n",
       "L 119.429672 93.034831 \r\n",
       "L 121.517777 89.91157 \r\n",
       "L 123.606924 87.075042 \r\n",
       "L 125.695706 84.526435 \r\n",
       "L 127.782725 82.266545 \r\n",
       "L 129.866588 80.295779 \r\n",
       "L 131.945918 78.614153 \r\n",
       "L 134.019348 77.2213 \r\n",
       "L 136.085535 76.116467 \r\n",
       "L 138.14315 75.298528 \r\n",
       "L 140.190893 74.765986 \r\n",
       "L 142.227487 74.516981 \r\n",
       "L 144.251683 74.549301 \r\n",
       "L 146.262265 74.860388 \r\n",
       "L 148.258047 75.447353 \r\n",
       "L 150.237881 76.306983 \r\n",
       "L 152.200654 77.435759 \r\n",
       "L 154.145292 78.829865 \r\n",
       "L 156.070761 80.485201 \r\n",
       "L 157.976069 82.397402 \r\n",
       "L 159.860267 84.561849 \r\n",
       "L 161.722449 86.973686 \r\n",
       "L 163.561754 89.627835 \r\n",
       "L 165.377367 92.519012 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 112.067624 123.992157 \r\n",
       "L 114.121497 118.915034 \r\n",
       "L 116.184598 114.106056 \r\n",
       "L 118.255554 109.569064 \r\n",
       "L 120.332979 105.307539 \r\n",
       "L 122.415474 101.324596 \r\n",
       "L 124.501635 97.622977 \r\n",
       "L 126.59005 94.205043 \r\n",
       "L 128.679309 91.07277 \r\n",
       "L 130.768002 88.227742 \r\n",
       "L 132.854722 85.671156 \r\n",
       "L 134.938074 83.403814 \r\n",
       "L 137.016672 81.426125 \r\n",
       "L 139.089142 79.73811 \r\n",
       "L 141.15413 78.3394 \r\n",
       "L 143.210302 77.229243 \r\n",
       "L 145.256344 76.40651 \r\n",
       "L 147.29097 75.869698 \r\n",
       "L 149.312921 75.616943 \r\n",
       "L 151.320968 75.646022 \r\n",
       "L 153.313915 75.954371 \r\n",
       "L 155.290599 76.539089 \r\n",
       "L 157.249897 77.396953 \r\n",
       "L 159.190722 78.524428 \r\n",
       "L 161.112027 79.917686 \r\n",
       "L 163.012808 81.57261 \r\n",
       "L 164.892104 83.484821 \r\n",
       "L 166.748997 85.64968 \r\n",
       "L 168.582613 88.062315 \r\n",
       "L 170.392128 90.717629 \r\n",
       "L 172.17676 93.610317 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 119.194682 124.142048 \r\n",
       "L 121.262751 119.046961 \r\n",
       "L 123.338492 114.220916 \r\n",
       "L 125.420516 109.667781 \r\n",
       "L 127.507422 105.391061 \r\n",
       "L 129.597799 101.393896 \r\n",
       "L 131.690231 97.679044 \r\n",
       "L 133.783298 94.248885 \r\n",
       "L 135.875585 91.105408 \r\n",
       "L 137.965675 88.25021 \r\n",
       "L 140.052162 85.684495 \r\n",
       "L 142.13365 83.409071 \r\n",
       "L 144.208756 81.424351 \r\n",
       "L 146.276111 79.730354 \r\n",
       "L 148.334368 78.326711 \r\n",
       "L 150.382201 77.212662 \r\n",
       "L 152.418309 76.38707 \r\n",
       "L 154.44142 75.848423 \r\n",
       "L 156.450288 75.594842 \r\n",
       "L 158.443704 75.62409 \r\n",
       "L 160.42049 75.933584 \r\n",
       "L 162.379507 76.520402 \r\n",
       "L 164.319652 77.381299 \r\n",
       "L 166.239864 78.512717 \r\n",
       "L 168.139124 79.9108 \r\n",
       "L 170.016455 81.571405 \r\n",
       "L 171.870926 83.490121 \r\n",
       "L 173.701649 85.66228 \r\n",
       "L 175.507786 88.082978 \r\n",
       "L 177.288542 90.747083 \r\n",
       "L 179.043173 93.649259 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 126.360695 123.18379 \r\n",
       "L 128.444986 118.064903 \r\n",
       "L 130.535383 113.216449 \r\n",
       "L 132.630476 108.64233 \r\n",
       "L 134.728848 104.346086 \r\n",
       "L 136.829071 100.330883 \r\n",
       "L 138.929715 96.599507 \r\n",
       "L 141.029352 93.154356 \r\n",
       "L 143.126555 89.997438 \r\n",
       "L 145.219902 87.130364 \r\n",
       "L 147.307982 84.554346 \r\n",
       "L 149.389397 82.270198 \r\n",
       "L 151.462764 80.278336 \r\n",
       "L 153.526717 78.578775 \r\n",
       "L 155.579915 77.17114 \r\n",
       "L 157.621038 76.054662 \r\n",
       "L 159.648796 75.228191 \r\n",
       "L 161.661926 74.690196 \r\n",
       "L 163.659198 74.438777 \r\n",
       "L 165.639419 74.471675 \r\n",
       "L 167.601428 74.786278 \r\n",
       "L 169.544107 75.379635 \r\n",
       "L 171.466375 76.248467 \r\n",
       "L 173.367196 77.38918 \r\n",
       "L 175.245576 78.797879 \r\n",
       "L 177.100565 80.470382 \r\n",
       "L 178.931261 82.402233 \r\n",
       "L 180.736809 84.588721 \r\n",
       "L 182.5164 87.024894 \r\n",
       "L 184.269275 89.705572 \r\n",
       "L 185.994723 92.62537 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 133.585763 121.099775 \r\n",
       "L 135.688394 115.951091 \r\n",
       "L 137.795545 111.074737 \r\n",
       "L 139.905782 106.474663 \r\n",
       "L 142.017668 102.154447 \r\n",
       "L 144.129756 98.117291 \r\n",
       "L 146.240601 94.36601 \r\n",
       "L 148.34876 90.903031 \r\n",
       "L 150.452794 87.730379 \r\n",
       "L 152.551274 84.849682 \r\n",
       "L 154.64278 82.262164 \r\n",
       "L 156.72591 79.968642 \r\n",
       "L 158.799279 77.969533 \r\n",
       "L 160.861523 76.264849 \r\n",
       "L 162.911302 74.854203 \r\n",
       "L 164.947302 73.736813 \r\n",
       "L 166.96824 72.911508 \r\n",
       "L 168.972864 72.376735 \r\n",
       "L 170.959958 72.130565 \r\n",
       "L 172.92834 72.170706 \r\n",
       "L 174.876869 72.494508 \r\n",
       "L 176.804445 73.098981 \r\n",
       "L 178.710008 73.9808 \r\n",
       "L 180.592546 75.136324 \r\n",
       "L 182.451089 76.561606 \r\n",
       "L 184.284715 78.25241 \r\n",
       "L 186.092552 80.204223 \r\n",
       "L 187.873772 82.412275 \r\n",
       "L 189.627602 84.871551 \r\n",
       "L 191.353314 87.576809 \r\n",
       "L 193.050234 90.522597 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 140.890712 117.865856 \r\n",
       "L 143.013908 112.681172 \r\n",
       "L 145.140006 107.771245 \r\n",
       "L 147.26755 103.140077 \r\n",
       "L 149.395074 98.791295 \r\n",
       "L 151.521114 94.728143 \r\n",
       "L 153.644205 90.953473 \r\n",
       "L 155.762886 87.469741 \r\n",
       "L 157.875704 84.278998 \r\n",
       "L 159.981218 81.382888 \r\n",
       "L 162.077999 78.782647 \r\n",
       "L 164.164638 76.479098 \r\n",
       "L 166.239746 74.472657 \r\n",
       "L 168.301957 72.763327 \r\n",
       "L 170.349932 71.350708 \r\n",
       "L 172.382361 70.234001 \r\n",
       "L 174.397966 69.412007 \r\n",
       "L 176.395504 68.883142 \r\n",
       "L 178.373769 68.645442 \r\n",
       "L 180.331595 68.696571 \r\n",
       "L 182.267855 69.033834 \r\n",
       "L 184.181467 69.654188 \r\n",
       "L 186.071394 70.554251 \r\n",
       "L 187.936643 71.730323 \r\n",
       "L 189.776272 73.17839 \r\n",
       "L 191.589385 74.894149 \r\n",
       "L 193.375137 76.873016 \r\n",
       "L 195.132734 79.110144 \r\n",
       "L 196.861432 81.600442 \r\n",
       "L 198.56054 84.338588 \r\n",
       "L 200.229416 87.319047 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 148.297308 113.45094 \r\n",
       "L 150.443414 108.223805 \r\n",
       "L 152.590766 103.274404 \r\n",
       "L 154.737878 98.606802 \r\n",
       "L 156.883258 94.224683 \r\n",
       "L 159.025417 90.13134 \r\n",
       "L 161.162868 86.329669 \r\n",
       "L 163.294129 82.822159 \r\n",
       "L 165.417731 79.610891 \r\n",
       "L 167.532218 76.697529 \r\n",
       "L 169.636149 74.08332 \r\n",
       "L 171.728106 71.769093 \r\n",
       "L 173.806693 69.755261 \r\n",
       "L 175.870539 68.041817 \r\n",
       "L 177.918306 66.628344 \r\n",
       "L 179.948684 65.514017 \r\n",
       "L 181.960401 64.697607 \r\n",
       "L 183.952221 64.17749 \r\n",
       "L 185.922946 63.951657 \r\n",
       "L 187.871423 64.01772 \r\n",
       "L 189.796542 64.372926 \r\n",
       "L 191.697236 65.014168 \r\n",
       "L 193.572489 65.937997 \r\n",
       "L 195.421331 67.140635 \r\n",
       "L 197.242844 68.617993 \r\n",
       "L 199.036159 70.365682 \r\n",
       "L 200.800461 72.379031 \r\n",
       "L 202.534985 74.653103 \r\n",
       "L 204.239022 77.182711 \r\n",
       "L 205.911914 79.962436 \r\n",
       "L 207.553058 82.986644 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 155.828489 107.816478 \r\n",
       "L 157.999989 102.54014 \r\n",
       "L 160.171028 97.545093 \r\n",
       "L 162.340088 92.835474 \r\n",
       "L 164.505647 88.415035 \r\n",
       "L 166.666187 84.287127 \r\n",
       "L 168.820195 80.454693 \r\n",
       "L 170.966167 76.920264 \r\n",
       "L 173.102613 73.685952 \r\n",
       "L 175.228059 70.753443 \r\n",
       "L 177.34105 68.123999 \r\n",
       "L 179.440157 65.798453 \r\n",
       "L 181.523973 63.777212 \r\n",
       "L 183.591123 62.060258 \r\n",
       "L 185.640264 60.647152 \r\n",
       "L 187.670087 59.537038 \r\n",
       "L 189.679323 58.728647 \r\n",
       "L 191.666741 58.22031 \r\n",
       "L 193.631153 58.009962 \r\n",
       "L 195.571417 58.095153 \r\n",
       "L 197.486435 58.473062 \r\n",
       "L 199.375161 59.140503 \r\n",
       "L 201.236596 60.093946 \r\n",
       "L 203.069794 61.329524 \r\n",
       "L 204.873861 62.843053 \r\n",
       "L 206.647955 64.630044 \r\n",
       "L 208.391291 66.685723 \r\n",
       "L 210.103136 69.005043 \r\n",
       "L 211.782814 71.582706 \r\n",
       "L 213.429704 74.413177 \r\n",
       "L 215.043239 77.490704 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 163.508622 100.915821 \r\n",
       "L 165.708158 95.583173 \r\n",
       "L 167.905466 90.535985 \r\n",
       "L 170.09899 85.778484 \r\n",
       "L 172.287174 81.314494 \r\n",
       "L 174.468467 77.147433 \r\n",
       "L 176.641328 73.280302 \r\n",
       "L 178.804226 69.715676 \r\n",
       "L 180.955647 66.455703 \r\n",
       "L 183.094098 63.502094 \r\n",
       "L 185.218106 60.856125 \r\n",
       "L 187.326225 58.518635 \r\n",
       "L 189.41704 56.490024 \r\n",
       "L 191.489166 54.770257 \r\n",
       "L 193.541256 53.358869 \r\n",
       "L 195.571999 52.254966 \r\n",
       "L 197.580125 51.457234 \r\n",
       "L 199.564411 50.963947 \r\n",
       "L 201.523674 50.772975 \r\n",
       "L 203.456785 50.881796 \r\n",
       "L 205.362659 51.287504 \r\n",
       "L 207.240265 51.986825 \r\n",
       "L 209.088627 52.97613 \r\n",
       "L 210.906819 54.251449 \r\n",
       "L 212.693973 55.808487 \r\n",
       "L 214.449276 57.642636 \r\n",
       "L 216.171972 59.749001 \r\n",
       "L 217.861362 62.122408 \r\n",
       "L 219.516805 64.757427 \r\n",
       "L 221.137716 67.648386 \r\n",
       "L 222.72357 70.789397 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 54.133789 83.013765 \r\n",
       "L 57.906249 87.717028 \r\n",
       "L 61.646625 92.119798 \r\n",
       "L 65.357599 96.226574 \r\n",
       "L 69.041795 100.041427 \r\n",
       "L 72.701787 103.568017 \r\n",
       "L 76.340106 106.809598 \r\n",
       "L 79.959242 109.769029 \r\n",
       "L 83.561656 112.448787 \r\n",
       "L 87.149779 114.850964 \r\n",
       "L 90.726022 116.977284 \r\n",
       "L 94.292779 118.829095 \r\n",
       "L 97.852434 120.407381 \r\n",
       "L 101.407365 121.712761 \r\n",
       "L 104.959949 122.745487 \r\n",
       "L 108.512571 123.505446 \r\n",
       "L 112.067624 123.992157 \r\n",
       "L 115.627517 124.204767 \r\n",
       "L 119.194682 124.142048 \r\n",
       "L 122.771578 123.802388 \r\n",
       "L 126.360695 123.18379 \r\n",
       "L 129.964565 122.283854 \r\n",
       "L 133.585763 121.099775 \r\n",
       "L 137.226916 119.628329 \r\n",
       "L 140.890712 117.865856 \r\n",
       "L 144.579901 115.808251 \r\n",
       "L 148.297308 113.45094 \r\n",
       "L 152.04584 110.78887 \r\n",
       "L 155.828489 107.816478 \r\n",
       "L 159.64835 104.527676 \r\n",
       "L 163.508622 100.915821 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 58.168328 73.018701 \r\n",
       "L 61.939387 77.749329 \r\n",
       "L 65.679222 82.176611 \r\n",
       "L 69.390511 86.305085 \r\n",
       "L 73.075882 90.138857 \r\n",
       "L 76.737909 93.681614 \r\n",
       "L 80.379127 96.936635 \r\n",
       "L 84.00203 99.906802 \r\n",
       "L 87.609084 102.594606 \r\n",
       "L 91.202726 105.002152 \r\n",
       "L 94.785375 107.13117 \r\n",
       "L 98.359433 108.983012 \r\n",
       "L 101.927294 110.558663 \r\n",
       "L 105.491347 111.858735 \r\n",
       "L 109.053982 112.883471 \r\n",
       "L 112.617596 113.632744 \r\n",
       "L 116.184598 114.106056 \r\n",
       "L 119.757414 114.302532 \r\n",
       "L 123.338492 114.220916 \r\n",
       "L 126.930311 113.859567 \r\n",
       "L 130.535383 113.216449 \r\n",
       "L 134.156261 112.289123 \r\n",
       "L 137.795545 111.074737 \r\n",
       "L 141.455888 109.570017 \r\n",
       "L 145.140006 107.771245 \r\n",
       "L 148.85068 105.674255 \r\n",
       "L 152.590766 103.274404 \r\n",
       "L 156.363205 100.566564 \r\n",
       "L 160.171028 97.545093 \r\n",
       "L 164.017368 94.203812 \r\n",
       "L 167.905466 90.535985 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 62.286647 64.152404 \r\n",
       "L 66.052737 68.904008 \r\n",
       "L 69.788521 73.349956 \r\n",
       "L 73.496675 77.494812 \r\n",
       "L 77.17982 81.342708 \r\n",
       "L 80.840529 84.897352 \r\n",
       "L 84.481334 88.162041 \r\n",
       "L 88.10473 91.13967 \r\n",
       "L 91.713182 93.83274 \r\n",
       "L 95.309131 96.243366 \r\n",
       "L 98.894996 98.37328 \r\n",
       "L 102.473187 100.223838 \r\n",
       "L 106.0461 101.796019 \r\n",
       "L 109.616134 103.090432 \r\n",
       "L 113.185685 104.107311 \r\n",
       "L 116.75716 104.846518 \r\n",
       "L 120.332979 105.307539 \r\n",
       "L 123.915579 105.489479 \r\n",
       "L 127.507422 105.391061 \r\n",
       "L 131.111002 105.010619 \r\n",
       "L 134.728848 104.346086 \r\n",
       "L 138.36353 103.394992 \r\n",
       "L 142.017668 102.154447 \r\n",
       "L 145.693937 100.621135 \r\n",
       "L 149.395074 98.791295 \r\n",
       "L 153.123887 96.660709 \r\n",
       "L 156.883258 94.224683 \r\n",
       "L 160.676157 91.478027 \r\n",
       "L 164.505647 88.415035 \r\n",
       "L 168.374893 85.029462 \r\n",
       "L 172.287174 81.314494 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 66.477777 56.441634 \r\n",
       "L 70.235355 61.207701 \r\n",
       "L 73.963603 65.666357 \r\n",
       "L 77.665186 69.822185 \r\n",
       "L 81.342717 73.679329 \r\n",
       "L 84.998762 77.241511 \r\n",
       "L 88.635845 80.512037 \r\n",
       "L 92.256456 83.493811 \r\n",
       "L 95.863058 86.189338 \r\n",
       "L 99.458087 88.600738 \r\n",
       "L 103.043963 90.729742 \r\n",
       "L 106.623095 92.577706 \r\n",
       "L 110.197881 94.145606 \r\n",
       "L 113.770721 95.434045 \r\n",
       "L 117.344017 96.443248 \r\n",
       "L 120.92018 97.173068 \r\n",
       "L 124.501635 97.622977 \r\n",
       "L 128.090828 97.792068 \r\n",
       "L 131.690231 97.679044 \r\n",
       "L 135.302346 97.28222 \r\n",
       "L 138.929715 96.599507 \r\n",
       "L 142.574922 95.628409 \r\n",
       "L 146.240601 94.36601 \r\n",
       "L 149.929444 92.808963 \r\n",
       "L 153.644205 90.953473 \r\n",
       "L 157.387711 88.795286 \r\n",
       "L 161.162868 86.329669 \r\n",
       "L 164.972666 83.551388 \r\n",
       "L 168.820195 80.454693 \r\n",
       "L 172.708646 77.033286 \r\n",
       "L 176.641328 73.280302 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 70.730341 49.906927 \r\n",
       "L 74.475909 54.680868 \r\n",
       "L 78.193176 59.146208 \r\n",
       "L 81.884789 63.307536 \r\n",
       "L 85.553348 67.169004 \r\n",
       "L 89.201406 70.734334 \r\n",
       "L 92.831477 74.006837 \r\n",
       "L 96.446042 76.989416 \r\n",
       "L 100.047554 79.684579 \r\n",
       "L 103.638444 82.094442 \r\n",
       "L 107.221125 84.220736 \r\n",
       "L 110.798002 86.064811 \r\n",
       "L 114.371471 87.627641 \r\n",
       "L 117.943929 88.90982 \r\n",
       "L 121.517777 89.91157 \r\n",
       "L 125.095428 90.632732 \r\n",
       "L 128.679309 91.07277 \r\n",
       "L 132.271869 91.230764 \r\n",
       "L 135.875585 91.105408 \r\n",
       "L 139.492964 90.695 \r\n",
       "L 143.126555 89.997438 \r\n",
       "L 146.77895 89.010209 \r\n",
       "L 150.452794 87.730379 \r\n",
       "L 154.15079 86.15458 \r\n",
       "L 157.875704 84.278998 \r\n",
       "L 161.630378 82.099354 \r\n",
       "L 165.417731 79.610891 \r\n",
       "L 169.240774 76.808351 \r\n",
       "L 173.102613 73.685952 \r\n",
       "L 177.006461 70.237369 \r\n",
       "L 180.955647 66.455703 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 75.032652 44.562414 \r\n",
       "L 78.762781 49.337608 \r\n",
       "L 82.46568 53.803584 \r\n",
       "L 86.143978 57.964925 \r\n",
       "L 89.800255 61.825776 \r\n",
       "L 93.437049 65.389856 \r\n",
       "L 97.056856 68.660469 \r\n",
       "L 100.662145 71.640515 \r\n",
       "L 104.255356 74.332495 \r\n",
       "L 107.838907 76.73852 \r\n",
       "L 111.415205 78.860316 \r\n",
       "L 114.986643 80.699227 \r\n",
       "L 118.555611 82.25622 \r\n",
       "L 122.1245 83.531885 \r\n",
       "L 125.695706 84.526435 \r\n",
       "L 129.271639 85.239707 \r\n",
       "L 132.854722 85.671156 \r\n",
       "L 136.447405 85.819857 \r\n",
       "L 140.052162 85.684495 \r\n",
       "L 143.671505 85.263361 \r\n",
       "L 147.307982 84.554346 \r\n",
       "L 150.964191 83.554928 \r\n",
       "L 154.64278 82.262164 \r\n",
       "L 158.346458 80.672678 \r\n",
       "L 162.077999 78.782647 \r\n",
       "L 165.840253 76.587783 \r\n",
       "L 169.636149 74.08332 \r\n",
       "L 173.468709 71.263988 \r\n",
       "L 177.34105 68.123999 \r\n",
       "L 181.256401 64.657014 \r\n",
       "L 185.218106 60.856125 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 79.372827 40.415722 \r\n",
       "L 83.084169 45.185572 \r\n",
       "L 86.769392 49.646154 \r\n",
       "L 90.431101 53.802038 \r\n",
       "L 94.071854 57.657352 \r\n",
       "L 97.694164 61.215803 \r\n",
       "L 101.300512 64.480683 \r\n",
       "L 104.893345 67.454877 \r\n",
       "L 108.475087 70.140878 \r\n",
       "L 112.048142 72.540787 \r\n",
       "L 115.6149 74.65632 \r\n",
       "L 119.177744 76.488814 \r\n",
       "L 122.739052 78.039229 \r\n",
       "L 126.301204 79.308148 \r\n",
       "L 129.866588 80.295779 \r\n",
       "L 133.437605 81.001954 \r\n",
       "L 137.016672 81.426125 \r\n",
       "L 140.606231 81.567363 \r\n",
       "L 144.208756 81.424351 \r\n",
       "L 147.82675 80.995379 \r\n",
       "L 151.462764 80.278336 \r\n",
       "L 155.119391 79.270701 \r\n",
       "L 158.799279 77.969533 \r\n",
       "L 162.505139 76.371459 \r\n",
       "L 166.239746 74.472657 \r\n",
       "L 170.005953 72.268845 \r\n",
       "L 173.806693 69.755261 \r\n",
       "L 177.644991 66.926644 \r\n",
       "L 181.523973 63.777212 \r\n",
       "L 185.446872 60.300638 \r\n",
       "L 189.41704 56.490024 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 83.738887 37.467973 \r\n",
       "L 87.428195 42.225945 \r\n",
       "L 91.09253 46.675169 \r\n",
       "L 94.734466 50.820184 \r\n",
       "L 98.356534 54.665097 \r\n",
       "L 101.961221 58.213591 \r\n",
       "L 105.550983 61.468936 \r\n",
       "L 109.128246 64.434004 \r\n",
       "L 112.695413 67.111268 \r\n",
       "L 116.254868 69.502816 \r\n",
       "L 119.808983 71.610353 \r\n",
       "L 123.360124 73.435206 \r\n",
       "L 126.910653 74.978325 \r\n",
       "L 130.462937 76.240289 \r\n",
       "L 134.019348 77.2213 \r\n",
       "L 137.582277 77.921186 \r\n",
       "L 141.15413 78.3394 \r\n",
       "L 144.73734 78.475013 \r\n",
       "L 148.334368 78.326711 \r\n",
       "L 151.947713 77.892788 \r\n",
       "L 155.579915 77.17114 \r\n",
       "L 159.233564 76.159255 \r\n",
       "L 162.911302 74.854203 \r\n",
       "L 166.615834 73.252622 \r\n",
       "L 170.349932 71.350708 \r\n",
       "L 174.116445 69.144196 \r\n",
       "L 177.918306 66.628344 \r\n",
       "L 181.758538 63.797915 \r\n",
       "L 185.640264 60.647152 \r\n",
       "L 189.566719 57.169759 \r\n",
       "L 193.541256 53.358869 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 88.118869 35.713859 \r\n",
       "L 91.783017 40.453536 \r\n",
       "L 95.423361 44.885539 \r\n",
       "L 99.042444 49.014371 \r\n",
       "L 102.642764 52.844103 \r\n",
       "L 106.226781 56.378389 \r\n",
       "L 109.79692 59.620473 \r\n",
       "L 113.355581 62.5732 \r\n",
       "L 116.905143 65.239025 \r\n",
       "L 120.447965 67.620016 \r\n",
       "L 123.9864 69.717864 \r\n",
       "L 127.522789 71.533883 \r\n",
       "L 131.059478 73.069014 \r\n",
       "L 134.598812 74.32383 \r\n",
       "L 138.14315 75.298528 \r\n",
       "L 141.694864 75.992937 \r\n",
       "L 145.256344 76.40651 \r\n",
       "L 148.83001 76.538323 \r\n",
       "L 152.418309 76.38707 \r\n",
       "L 156.023729 75.951056 \r\n",
       "L 159.648796 75.228191 \r\n",
       "L 163.296089 74.215978 \r\n",
       "L 166.96824 72.911508 \r\n",
       "L 170.667945 71.311444 \r\n",
       "L 174.397966 69.412007 \r\n",
       "L 178.161143 67.208963 \r\n",
       "L 181.960401 64.697607 \r\n",
       "L 185.798756 61.872739 \r\n",
       "L 189.679323 58.728647 \r\n",
       "L 193.605331 55.259082 \r\n",
       "L 197.580125 51.457234 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 92.500935 35.141811 \r\n",
       "L 96.136924 39.856932 \r\n",
       "L 99.750301 44.265997 \r\n",
       "L 103.343571 48.373461 \r\n",
       "L 106.919195 52.183353 \r\n",
       "L 110.4796 55.699287 \r\n",
       "L 114.027179 58.924474 \r\n",
       "L 117.564303 61.86173 \r\n",
       "L 121.093321 64.513482 \r\n",
       "L 124.616566 66.881779 \r\n",
       "L 128.136362 68.968291 \r\n",
       "L 131.655029 70.77432 \r\n",
       "L 135.174887 72.300797 \r\n",
       "L 138.698261 73.548285 \r\n",
       "L 142.227487 74.516981 \r\n",
       "L 145.764916 75.206715 \r\n",
       "L 149.312921 75.616943 \r\n",
       "L 152.873902 75.74675 \r\n",
       "L 156.450288 75.594842 \r\n",
       "L 160.04455 75.159541 \r\n",
       "L 163.659198 74.438777 \r\n",
       "L 167.296795 73.430079 \r\n",
       "L 170.959958 72.130565 \r\n",
       "L 174.651366 70.536933 \r\n",
       "L 178.373769 68.645442 \r\n",
       "L 182.129993 66.451902 \r\n",
       "L 185.922946 63.951657 \r\n",
       "L 189.755632 61.13956 \r\n",
       "L 193.631153 58.009962 \r\n",
       "L 197.552723 54.55668 \r\n",
       "L 201.523674 50.772975 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 96.873466 35.734237 \r\n",
       "L 100.478446 40.41874 \r\n",
       "L 104.062017 44.799331 \r\n",
       "L 107.626644 48.880407 \r\n",
       "L 111.17475 52.665947 \r\n",
       "L 114.708722 56.159517 \r\n",
       "L 118.230921 59.36429 \r\n",
       "L 121.743681 62.283043 \r\n",
       "L 125.24932 64.918175 \r\n",
       "L 128.750141 67.271709 \r\n",
       "L 132.248438 69.345295 \r\n",
       "L 135.746502 71.140217 \r\n",
       "L 139.246627 72.657396 \r\n",
       "L 142.751111 73.897388 \r\n",
       "L 146.262265 74.860388 \r\n",
       "L 149.782416 75.546227 \r\n",
       "L 153.313915 75.954371 \r\n",
       "L 156.859136 76.083916 \r\n",
       "L 160.42049 75.933584 \r\n",
       "L 164.000424 75.501719 \r\n",
       "L 167.601428 74.786278 \r\n",
       "L 171.226044 73.784822 \r\n",
       "L 174.876869 72.494508 \r\n",
       "L 178.556563 70.912076 \r\n",
       "L 182.267855 69.033834 \r\n",
       "L 186.013551 66.85565 \r\n",
       "L 189.796542 64.372926 \r\n",
       "L 193.619809 61.580588 \r\n",
       "L 197.486435 58.473062 \r\n",
       "L 201.399615 55.044249 \r\n",
       "L 205.362659 51.287504 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 101.225166 37.467842 \r\n",
       "L 104.796438 42.115903 \r\n",
       "L 108.347515 46.462699 \r\n",
       "L 111.880813 50.512563 \r\n",
       "L 115.398714 54.269412 \r\n",
       "L 118.903567 57.736762 \r\n",
       "L 122.39769 60.917736 \r\n",
       "L 125.883382 63.815073 \r\n",
       "L 129.362926 66.431136 \r\n",
       "L 132.838588 68.767918 \r\n",
       "L 136.312633 70.827047 \r\n",
       "L 139.787318 72.609789 \r\n",
       "L 143.264906 74.117051 \r\n",
       "L 146.747666 75.349385 \r\n",
       "L 150.237881 76.306983 \r\n",
       "L 153.737851 76.989681 \r\n",
       "L 157.249897 77.396953 \r\n",
       "L 160.77637 77.527911 \r\n",
       "L 164.319652 77.381299 \r\n",
       "L 167.882165 76.955488 \r\n",
       "L 171.466375 76.248467 \r\n",
       "L 175.074799 75.257837 \r\n",
       "L 178.710008 73.9808 \r\n",
       "L 182.374638 72.414148 \r\n",
       "L 186.071394 70.554251 \r\n",
       "L 189.803056 68.397041 \r\n",
       "L 193.572489 65.937997 \r\n",
       "L 197.38265 63.172126 \r\n",
       "L 201.236596 60.093946 \r\n",
       "L 205.137494 56.697459 \r\n",
       "L 209.088627 52.97613 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 105.545142 40.314003 \r\n",
       "L 109.080176 44.920067 \r\n",
       "L 112.596226 49.227995 \r\n",
       "L 116.095663 53.242044 \r\n",
       "L 119.580822 56.966064 \r\n",
       "L 123.054007 60.403512 \r\n",
       "L 126.517497 63.557458 \r\n",
       "L 129.973549 66.430596 \r\n",
       "L 133.424407 69.02525 \r\n",
       "L 136.872302 71.34338 \r\n",
       "L 140.319459 73.386589 \r\n",
       "L 143.768104 75.156122 \r\n",
       "L 147.220464 76.652876 \r\n",
       "L 150.678777 77.877393 \r\n",
       "L 154.145292 78.829865 \r\n",
       "L 157.622278 79.510133 \r\n",
       "L 161.112027 79.917686 \r\n",
       "L 164.616858 80.051651 \r\n",
       "L 168.139124 79.9108 \r\n",
       "L 171.681217 79.493533 \r\n",
       "L 175.245576 78.797879 \r\n",
       "L 178.834685 77.821485 \r\n",
       "L 182.451089 76.561606 \r\n",
       "L 186.097393 75.015095 \r\n",
       "L 189.776272 73.17839 \r\n",
       "L 193.490478 71.047501 \r\n",
       "L 197.242844 68.617993 \r\n",
       "L 201.036297 65.88497 \r\n",
       "L 204.873861 62.843053 \r\n",
       "L 208.758669 59.486361 \r\n",
       "L 212.693973 55.808487 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 109.822991 44.239201 \r\n",
       "L 113.319427 48.798016 \r\n",
       "L 116.798086 53.062274 \r\n",
       "L 120.261289 57.036151 \r\n",
       "L 123.711322 60.723425 \r\n",
       "L 127.150444 64.127484 \r\n",
       "L 130.580887 67.251343 \r\n",
       "L 134.004868 70.097644 \r\n",
       "L 137.424588 72.668669 \r\n",
       "L 140.842237 74.966343 \r\n",
       "L 144.260003 76.99224 \r\n",
       "L 147.680072 78.747586 \r\n",
       "L 151.104637 80.233261 \r\n",
       "L 154.535898 81.449803 \r\n",
       "L 157.976069 82.397402 \r\n",
       "L 161.427386 83.075908 \r\n",
       "L 164.892104 83.484821 \r\n",
       "L 168.372511 83.623292 \r\n",
       "L 171.870926 83.490121 \r\n",
       "L 175.389708 83.083743 \r\n",
       "L 178.931261 82.402233 \r\n",
       "L 182.498039 81.443288 \r\n",
       "L 186.092552 80.204223 \r\n",
       "L 189.717371 78.681961 \r\n",
       "L 193.375137 76.873016 \r\n",
       "L 197.068568 74.773485 \r\n",
       "L 200.800461 72.379031 \r\n",
       "L 204.573706 69.684864 \r\n",
       "L 208.391291 66.685723 \r\n",
       "L 212.256309 63.375857 \r\n",
       "L 216.171972 59.749001 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 114.048865 49.205501 \r\n",
       "L 117.50452 53.712137 \r\n",
       "L 120.943595 57.928221 \r\n",
       "L 124.368358 61.857837 \r\n",
       "L 127.781044 65.504683 \r\n",
       "L 131.183863 68.872077 \r\n",
       "L 134.579001 71.962971 \r\n",
       "L 137.968627 74.779952 \r\n",
       "L 141.3549 77.325258 \r\n",
       "L 144.739968 79.600774 \r\n",
       "L 148.125975 81.608045 \r\n",
       "L 151.51507 83.348274 \r\n",
       "L 154.909403 84.822328 \r\n",
       "L 158.311137 86.030734 \r\n",
       "L 161.722449 86.973686 \r\n",
       "L 165.145534 87.651041 \r\n",
       "L 168.582613 88.062315 \r\n",
       "L 172.035936 88.206684 \r\n",
       "L 175.507786 88.082978 \r\n",
       "L 179.000485 87.689674 \r\n",
       "L 182.5164 87.024894 \r\n",
       "L 186.057948 86.086392 \r\n",
       "L 189.627602 84.871551 \r\n",
       "L 193.227896 83.377367 \r\n",
       "L 196.861432 81.600442 \r\n",
       "L 200.530888 79.536967 \r\n",
       "L 204.239022 77.182711 \r\n",
       "L 207.988683 74.533 \r\n",
       "L 211.782814 71.582706 \r\n",
       "L 215.624467 68.326218 \r\n",
       "L 219.516805 64.757427 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "    <path clip-path=\"url(#paadc75025e)\" d=\"M 118.213531 55.171053 \r\n",
       "L 121.626404 59.620929 \r\n",
       "L 125.023878 63.784647 \r\n",
       "L 128.408166 67.666195 \r\n",
       "L 131.781449 71.269186 \r\n",
       "L 135.145887 74.596862 \r\n",
       "L 138.503618 77.652106 \r\n",
       "L 141.856762 80.437451 \r\n",
       "L 145.207431 82.955082 \r\n",
       "L 148.557729 85.206847 \r\n",
       "L 151.909757 87.194258 \r\n",
       "L 155.265619 88.918493 \r\n",
       "L 158.627425 90.380404 \r\n",
       "L 161.997296 91.580512 \r\n",
       "L 165.377367 92.519012 \r\n",
       "L 168.769795 93.195769 \r\n",
       "L 172.17676 93.610317 \r\n",
       "L 175.600471 93.761859 \r\n",
       "L 179.043173 93.649259 \r\n",
       "L 182.507148 93.271038 \r\n",
       "L 185.994723 92.62537 \r\n",
       "L 189.508274 91.710072 \r\n",
       "L 193.050234 90.522597 \r\n",
       "L 196.623094 89.060025 \r\n",
       "L 200.229416 87.319047 \r\n",
       "L 203.871832 85.295959 \r\n",
       "L 207.553058 82.986644 \r\n",
       "L 211.275894 80.386556 \r\n",
       "L 215.043239 77.490704 \r\n",
       "L 218.858093 74.293632 \r\n",
       "L 222.72357 70.789397 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_13\">\r\n",
       "    <defs>\r\n",
       "     <path d=\"M -3 3 \r\n",
       "L 3 -3 \r\n",
       "M -3 -3 \r\n",
       "L 3 3 \r\n",
       "\" id=\"m689088a1fe\" style=\"stroke:#ff0000;\"/>\r\n",
       "    </defs>\r\n",
       "    <g clip-path=\"url(#paadc75025e)\">\r\n",
       "     <use style=\"fill:#ff0000;stroke:#ff0000;\" x=\"139.643243\" xlink:href=\"#m689088a1fe\" y=\"76.813514\"/>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"paadc75025e\">\r\n",
       "   <rect height=\"135.9\" width=\"251.1\" x=\"10.7\" y=\"10.7\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 324x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x, y = np.mgrid[-1: 1: 31j, -1: 1: 31j]\n",
    "z = x**2 - y**2\n",
    "\n",
    "ax = dl_utils.plt.figure().add_subplot(111, projection='3d')\n",
    "ax.plot_wireframe(x, y, z, **{'rstride': 2, 'cstride': 2})\n",
    "ax.plot([0], [0], [0], 'rx')\n",
    "ticks = [-1,  0, 1]\n",
    "dl_utils.plt.xticks(ticks)\n",
    "dl_utils.plt.yticks(ticks)\n",
    "ax.set_zticks(ticks)\n",
    "dl_utils.plt.xlabel('x')\n",
    "dl_utils.plt.ylabel('y');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在图的鞍点位置，目标函数在$x$轴方向上是局部最小值，但在$y$轴方向上是局部最大值。\n",
    "\n",
    "假设一个函数的输入为$k$维向量，输出为标量，那么它的海森矩阵（Hessian matrix）有$k$个特征值。该函数在梯度为0的位置上可能是局部最小值、局部最大值或者鞍点。\n",
    "\n",
    "- 当函数的海森矩阵在梯度为零的位置上的特征值全为正时，该函数得到局部最小值。\n",
    "- 当函数的海森矩阵在梯度为零的位置上的特征值全为负时，该函数得到局部最大值。\n",
    "- 当函数的海森矩阵在梯度为零的位置上的特征值有正有负时，该函数得到鞍点。\n",
    "\n",
    "随机矩阵理论告诉我们，对于一个大的高斯随机矩阵来说，任一特征值是正或者是负的概率都是0.5 [1]。那么，以上第一种情况的概率为 $0.5^k$。由于深度学习模型参数通常都是高维的（$k$很大），目标函数的鞍点通常比局部最小值更常见。\n",
    "\n",
    "在深度学习中，虽然找到目标函数的全局最优解很难，但这并非必要。我们将在本章接下来的几节中逐一介绍深度学习中常用的优化算法，它们在很多实际问题中都能够训练出十分有效的深度学习模型。\n",
    "\n",
    "## 小结\n",
    "\n",
    "- 由于优化算法的目标函数通常是一个基于训练数据集的损失函数，优化的目标在于降低训练误差。\n",
    "- 由于深度学习模型参数通常都是高维的，目标函数的鞍点通常比局部最小值更常见。\n",
    "\n",
    "## 参考文献\n",
    "\n",
    "[1] Wigner, E. P. (1958). On the distribution of the roots of certain symmetric matrices. Annals of Mathematics, 325-327."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 梯度下降和随机梯度下降\n",
    "\n",
    "绍梯度下降（gradient descent）,虽然梯度下降在深度学习中很少被直接使用，但理解\n",
    "\n",
    "**梯度的意义**以及**沿着梯度反方向更新自变量**可能降低目标函数值的原因是学习后续优化算法的基础。随后，我们将引出随机梯度下降（stochastic gradient descent）。\n",
    "\n",
    "##  一维梯度下降\n",
    "\n",
    "我们先以简单的一维梯度下降为例，解释梯度下降算法可能降低目标函数值的原因。假设连续可导的函数$f: \\mathbb{R} \\rightarrow \\mathbb{R}$的输入和输出都是标量。给定绝对值足够小的数$\\epsilon$，根据泰勒展开公式，我们得到以下的近似：\n",
    "\n",
    "$$f(x + \\epsilon) \\approx f(x) + \\epsilon f'(x) .$$\n",
    "\n",
    "这里$f'(x)$是函数$f$在$x$处的梯度。一维函数的梯度是一个标量，也称导数。\n",
    "\n",
    "接下来，找到一个常数$\\eta > 0$，使得$\\left|\\eta f'(x)\\right|$足够小，那么可以将$\\epsilon$替换为$-\\eta f'(x)$并得到\n",
    "\n",
    "$$f(x - \\eta f'(x)) \\approx f(x) -  \\eta f'(x)^2.$$\n",
    "\n",
    "如果导数$f'(x) \\neq 0$，那么$\\eta f'(x)^2>0$，所以\n",
    "\n",
    "$$f(x - \\eta f'(x)) \\lesssim f(x).$$\n",
    "\n",
    "这意味着，如果通过\n",
    "\n",
    "$$x \\leftarrow x - \\eta f'(x)$$\n",
    "\n",
    "来迭代$x$，函数$f(x)$的值可能会降低。因此在梯度下降中，我们先选取一个初始值$x$和常数$\\eta > 0$，然后不断通过上式来迭代$x$，直到达到停止条件，例如$f'(x)^2$的值已足够小或迭代次数已达到某个值。\n",
    "\n",
    "下面我们以目标函数$f(x)=x^2$为例来看一看梯度下降是如何工作的。虽然我们知道最小化$f(x)$的解为$x=0$，这里依然使用这个简单函数来观察$x$是如何被迭代的。首先，导入本节实验所需的包或模块。\n",
    "\n",
    "接下来使用$x=10$作为初始值，并设$\\eta=0.2$。使用梯度下降对$x$迭代10次，可见最终$x$的值较接近最优解。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 0.06046617599999997\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "def gd(eta):\n",
    "    x = 10\n",
    "    results = [x]\n",
    "    for i in range(10):\n",
    "        x -= eta * 2 * x  # f(x) = x * x的导数为f'(x) = 2 * x\n",
    "        results.append(x)\n",
    "    print('epoch 10, x:', x)\n",
    "    return results\n",
    "\n",
    "res = gd(0.2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\r\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 252.965625 184.15625\" width=\"252.965625pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 184.15625 \r\n",
       "L 252.965625 184.15625 \r\n",
       "L 252.965625 -0 \r\n",
       "L 0 -0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 46.965625 146.6 \r\n",
       "L 242.265625 146.6 \r\n",
       "L 242.265625 10.7 \r\n",
       "L 46.965625 10.7 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m074b9a751c\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.842898\" xlink:href=\"#m074b9a751c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- −10 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\"/>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\"/>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(45.290554 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"100.229261\" xlink:href=\"#m074b9a751c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- −5 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.796875 72.90625 \r\n",
       "L 49.515625 72.90625 \r\n",
       "L 49.515625 64.59375 \r\n",
       "L 19.828125 64.59375 \r\n",
       "L 19.828125 46.734375 \r\n",
       "Q 21.96875 47.46875 24.109375 47.828125 \r\n",
       "Q 26.265625 48.1875 28.421875 48.1875 \r\n",
       "Q 40.625 48.1875 47.75 41.5 \r\n",
       "Q 54.890625 34.8125 54.890625 23.390625 \r\n",
       "Q 54.890625 11.625 47.5625 5.09375 \r\n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \r\n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \r\n",
       "Q 12.796875 0.140625 7.71875 1.703125 \r\n",
       "L 7.71875 11.625 \r\n",
       "Q 12.109375 9.234375 16.796875 8.0625 \r\n",
       "Q 21.484375 6.890625 26.703125 6.890625 \r\n",
       "Q 35.15625 6.890625 40.078125 11.328125 \r\n",
       "Q 45.015625 15.765625 45.015625 23.390625 \r\n",
       "Q 45.015625 31 40.078125 35.4375 \r\n",
       "Q 35.15625 39.890625 26.703125 39.890625 \r\n",
       "Q 22.75 39.890625 18.8125 39.015625 \r\n",
       "Q 14.890625 38.140625 10.796875 36.28125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-35\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(92.858168 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"144.615625\" xlink:href=\"#m074b9a751c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(141.434375 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"189.001989\" xlink:href=\"#m074b9a751c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 5 -->\r\n",
       "      <g transform=\"translate(185.820739 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"233.388352\" xlink:href=\"#m074b9a751c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 10 -->\r\n",
       "      <g transform=\"translate(227.025852 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_6\">\r\n",
       "     <!-- x -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 54.890625 54.6875 \r\n",
       "L 35.109375 28.078125 \r\n",
       "L 55.90625 0 \r\n",
       "L 45.3125 0 \r\n",
       "L 29.390625 21.484375 \r\n",
       "L 13.484375 0 \r\n",
       "L 2.875 0 \r\n",
       "L 24.125 28.609375 \r\n",
       "L 4.6875 54.6875 \r\n",
       "L 15.28125 54.6875 \r\n",
       "L 29.78125 35.203125 \r\n",
       "L 44.28125 54.6875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(141.65625 174.876562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m3e4ffecb78\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.965625\" xlink:href=\"#m3e4ffecb78\" y=\"140.422727\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(33.603125 144.221946)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.965625\" xlink:href=\"#m3e4ffecb78\" y=\"115.713636\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 20 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(27.240625 119.512855)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.965625\" xlink:href=\"#m3e4ffecb78\" y=\"91.004545\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 40 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(27.240625 94.803764)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.965625\" xlink:href=\"#m3e4ffecb78\" y=\"66.295455\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- 60 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 33.015625 40.375 \r\n",
       "Q 26.375 40.375 22.484375 35.828125 \r\n",
       "Q 18.609375 31.296875 18.609375 23.390625 \r\n",
       "Q 18.609375 15.53125 22.484375 10.953125 \r\n",
       "Q 26.375 6.390625 33.015625 6.390625 \r\n",
       "Q 39.65625 6.390625 43.53125 10.953125 \r\n",
       "Q 47.40625 15.53125 47.40625 23.390625 \r\n",
       "Q 47.40625 31.296875 43.53125 35.828125 \r\n",
       "Q 39.65625 40.375 33.015625 40.375 \r\n",
       "z\r\n",
       "M 52.59375 71.296875 \r\n",
       "L 52.59375 62.3125 \r\n",
       "Q 48.875 64.0625 45.09375 64.984375 \r\n",
       "Q 41.3125 65.921875 37.59375 65.921875 \r\n",
       "Q 27.828125 65.921875 22.671875 59.328125 \r\n",
       "Q 17.53125 52.734375 16.796875 39.40625 \r\n",
       "Q 19.671875 43.65625 24.015625 45.921875 \r\n",
       "Q 28.375 48.1875 33.59375 48.1875 \r\n",
       "Q 44.578125 48.1875 50.953125 41.515625 \r\n",
       "Q 57.328125 34.859375 57.328125 23.390625 \r\n",
       "Q 57.328125 12.15625 50.6875 5.359375 \r\n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \r\n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \r\n",
       "Q 6.984375 17.96875 6.984375 36.375 \r\n",
       "Q 6.984375 53.65625 15.1875 63.9375 \r\n",
       "Q 23.390625 74.21875 37.203125 74.21875 \r\n",
       "Q 40.921875 74.21875 44.703125 73.484375 \r\n",
       "Q 48.484375 72.75 52.59375 71.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-36\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(27.240625 70.094673)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.965625\" xlink:href=\"#m3e4ffecb78\" y=\"41.586364\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- 80 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 34.625 \r\n",
       "Q 24.75 34.625 20.71875 30.859375 \r\n",
       "Q 16.703125 27.09375 16.703125 20.515625 \r\n",
       "Q 16.703125 13.921875 20.71875 10.15625 \r\n",
       "Q 24.75 6.390625 31.78125 6.390625 \r\n",
       "Q 38.8125 6.390625 42.859375 10.171875 \r\n",
       "Q 46.921875 13.96875 46.921875 20.515625 \r\n",
       "Q 46.921875 27.09375 42.890625 30.859375 \r\n",
       "Q 38.875 34.625 31.78125 34.625 \r\n",
       "z\r\n",
       "M 21.921875 38.8125 \r\n",
       "Q 15.578125 40.375 12.03125 44.71875 \r\n",
       "Q 8.5 49.078125 8.5 55.328125 \r\n",
       "Q 8.5 64.0625 14.71875 69.140625 \r\n",
       "Q 20.953125 74.21875 31.78125 74.21875 \r\n",
       "Q 42.671875 74.21875 48.875 69.140625 \r\n",
       "Q 55.078125 64.0625 55.078125 55.328125 \r\n",
       "Q 55.078125 49.078125 51.53125 44.71875 \r\n",
       "Q 48 40.375 41.703125 38.8125 \r\n",
       "Q 48.828125 37.15625 52.796875 32.3125 \r\n",
       "Q 56.78125 27.484375 56.78125 20.515625 \r\n",
       "Q 56.78125 9.90625 50.3125 4.234375 \r\n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.734375 -1.421875 13.25 4.234375 \r\n",
       "Q 6.78125 9.90625 6.78125 20.515625 \r\n",
       "Q 6.78125 27.484375 10.78125 32.3125 \r\n",
       "Q 14.796875 37.15625 21.921875 38.8125 \r\n",
       "z\r\n",
       "M 18.3125 54.390625 \r\n",
       "Q 18.3125 48.734375 21.84375 45.5625 \r\n",
       "Q 25.390625 42.390625 31.78125 42.390625 \r\n",
       "Q 38.140625 42.390625 41.71875 45.5625 \r\n",
       "Q 45.3125 48.734375 45.3125 54.390625 \r\n",
       "Q 45.3125 60.0625 41.71875 63.234375 \r\n",
       "Q 38.140625 66.40625 31.78125 66.40625 \r\n",
       "Q 25.390625 66.40625 21.84375 63.234375 \r\n",
       "Q 18.3125 60.0625 18.3125 54.390625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-38\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(27.240625 45.385582)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_6\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.965625\" xlink:href=\"#m3e4ffecb78\" y=\"16.877273\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <g transform=\"translate(20.878125 20.676491)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_13\">\r\n",
       "     <!-- f(x) -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 37.109375 75.984375 \r\n",
       "L 37.109375 68.5 \r\n",
       "L 28.515625 68.5 \r\n",
       "Q 23.6875 68.5 21.796875 66.546875 \r\n",
       "Q 19.921875 64.59375 19.921875 59.515625 \r\n",
       "L 19.921875 54.6875 \r\n",
       "L 34.71875 54.6875 \r\n",
       "L 34.71875 47.703125 \r\n",
       "L 19.921875 47.703125 \r\n",
       "L 19.921875 0 \r\n",
       "L 10.890625 0 \r\n",
       "L 10.890625 47.703125 \r\n",
       "L 2.296875 47.703125 \r\n",
       "L 2.296875 54.6875 \r\n",
       "L 10.890625 54.6875 \r\n",
       "L 10.890625 58.5 \r\n",
       "Q 10.890625 67.625 15.140625 71.796875 \r\n",
       "Q 19.390625 75.984375 28.609375 75.984375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-66\"/>\r\n",
       "      <path d=\"M 31 75.875 \r\n",
       "Q 24.46875 64.65625 21.28125 53.65625 \r\n",
       "Q 18.109375 42.671875 18.109375 31.390625 \r\n",
       "Q 18.109375 20.125 21.3125 9.0625 \r\n",
       "Q 24.515625 -2 31 -13.1875 \r\n",
       "L 23.1875 -13.1875 \r\n",
       "Q 15.875 -1.703125 12.234375 9.375 \r\n",
       "Q 8.59375 20.453125 8.59375 31.390625 \r\n",
       "Q 8.59375 42.28125 12.203125 53.3125 \r\n",
       "Q 15.828125 64.359375 23.1875 75.875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-28\"/>\r\n",
       "      <path d=\"M 8.015625 75.875 \r\n",
       "L 15.828125 75.875 \r\n",
       "Q 23.140625 64.359375 26.78125 53.3125 \r\n",
       "Q 30.421875 42.28125 30.421875 31.390625 \r\n",
       "Q 30.421875 20.453125 26.78125 9.375 \r\n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \r\n",
       "L 8.015625 -13.1875 \r\n",
       "Q 14.5 -2 17.703125 9.0625 \r\n",
       "Q 20.90625 20.125 20.90625 31.390625 \r\n",
       "Q 20.90625 42.671875 17.703125 53.65625 \r\n",
       "Q 14.5 64.65625 8.015625 75.875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-29\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798438 87.271094)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\r\n",
       "      <use x=\"35.205078\" xlink:href=\"#DejaVuSans-28\"/>\r\n",
       "      <use x=\"74.21875\" xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "      <use x=\"133.398438\" xlink:href=\"#DejaVuSans-29\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_12\">\r\n",
       "    <path clip-path=\"url(#pc0383faa75)\" d=\"M 55.842898 16.877273 \r\n",
       "L 60.281534 28.922955 \r\n",
       "L 64.72017 40.350909 \r\n",
       "L 69.158807 51.161136 \r\n",
       "L 73.597443 61.353636 \r\n",
       "L 78.03608 70.928409 \r\n",
       "L 82.474716 79.885455 \r\n",
       "L 86.025625 86.606327 \r\n",
       "L 89.576534 92.931855 \r\n",
       "L 93.127443 98.862036 \r\n",
       "L 96.678352 104.396873 \r\n",
       "L 100.229261 109.536364 \r\n",
       "L 103.78017 114.280509 \r\n",
       "L 107.33108 118.629309 \r\n",
       "L 110.881989 122.582764 \r\n",
       "L 114.432898 126.140873 \r\n",
       "L 117.983807 129.303636 \r\n",
       "L 120.646989 131.416264 \r\n",
       "L 123.31017 133.306509 \r\n",
       "L 125.973352 134.974373 \r\n",
       "L 128.636534 136.419855 \r\n",
       "L 131.299716 137.642955 \r\n",
       "L 133.962898 138.643673 \r\n",
       "L 136.62608 139.422009 \r\n",
       "L 139.289261 139.977964 \r\n",
       "L 141.952443 140.311536 \r\n",
       "L 144.615625 140.422727 \r\n",
       "L 147.278807 140.311536 \r\n",
       "L 149.941989 139.977964 \r\n",
       "L 152.60517 139.422009 \r\n",
       "L 155.268352 138.643673 \r\n",
       "L 157.931534 137.642955 \r\n",
       "L 160.594716 136.419855 \r\n",
       "L 163.257898 134.974373 \r\n",
       "L 165.92108 133.306509 \r\n",
       "L 168.584261 131.416264 \r\n",
       "L 171.247443 129.303636 \r\n",
       "L 173.910625 126.968627 \r\n",
       "L 177.461534 123.509355 \r\n",
       "L 181.012443 119.654736 \r\n",
       "L 184.563352 115.404773 \r\n",
       "L 188.114261 110.759464 \r\n",
       "L 191.66517 105.718809 \r\n",
       "L 195.21608 100.282809 \r\n",
       "L 198.766989 94.451464 \r\n",
       "L 202.317898 88.224773 \r\n",
       "L 205.868807 81.602736 \r\n",
       "L 209.419716 74.585355 \r\n",
       "L 213.858352 65.257673 \r\n",
       "L 218.296989 55.312264 \r\n",
       "L 222.735625 44.749127 \r\n",
       "L 227.174261 33.568264 \r\n",
       "L 231.612898 21.769673 \r\n",
       "L 232.500625 19.335827 \r\n",
       "L 232.500625 19.335827 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_13\">\r\n",
       "    <path clip-path=\"url(#pc0383faa75)\" d=\"M 233.388352 16.877273 \r\n",
       "L 197.879261 95.946364 \r\n",
       "L 176.573807 124.411236 \r\n",
       "L 163.790534 134.658591 \r\n",
       "L 156.12057 138.347638 \r\n",
       "L 151.518592 139.675695 \r\n",
       "L 148.757405 140.153796 \r\n",
       "L 147.100693 140.325912 \r\n",
       "L 146.106666 140.387874 \r\n",
       "L 145.51025 140.41018 \r\n",
       "L 145.1524 140.41821 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    <defs>\r\n",
       "     <path d=\"M 0 3 \r\n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \r\n",
       "C 2.683901 1.55874 3 0.795609 3 0 \r\n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \r\n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \r\n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \r\n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \r\n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \r\n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \r\n",
       "z\r\n",
       "\" id=\"m61e7e3c004\" style=\"stroke:#ff7f0e;\"/>\r\n",
       "    </defs>\r\n",
       "    <g clip-path=\"url(#pc0383faa75)\">\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"233.388352\" xlink:href=\"#m61e7e3c004\" y=\"16.877273\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"197.879261\" xlink:href=\"#m61e7e3c004\" y=\"95.946364\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"176.573807\" xlink:href=\"#m61e7e3c004\" y=\"124.411236\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"163.790534\" xlink:href=\"#m61e7e3c004\" y=\"134.658591\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"156.12057\" xlink:href=\"#m61e7e3c004\" y=\"138.347638\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"151.518592\" xlink:href=\"#m61e7e3c004\" y=\"139.675695\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"148.757405\" xlink:href=\"#m61e7e3c004\" y=\"140.153796\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"147.100693\" xlink:href=\"#m61e7e3c004\" y=\"140.325912\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"146.106666\" xlink:href=\"#m61e7e3c004\" y=\"140.387874\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"145.51025\" xlink:href=\"#m61e7e3c004\" y=\"140.41018\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"145.1524\" xlink:href=\"#m61e7e3c004\" y=\"140.41821\"/>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 46.965625 146.6 \r\n",
       "L 46.965625 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 242.265625 146.6 \r\n",
       "L 242.265625 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 46.965625 146.6 \r\n",
       "L 242.265625 146.6 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 46.965625 10.7 \r\n",
       "L 242.265625 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"pc0383faa75\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"46.965625\" y=\"10.7\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def show_trace(res):\n",
    "    n = max(abs(min(res)), abs(max(res)), 10)\n",
    "    f_line = np.arange(-n, n, 0.1)\n",
    "    dl_utils.set_figsize()\n",
    "    dl_utils.plt.plot(f_line, [x * x for x in f_line])\n",
    "    dl_utils.plt.plot(res, [x * x for x in res], '-o')\n",
    "    dl_utils.plt.xlabel('x')\n",
    "    dl_utils.plt.ylabel('f(x)')\n",
    "\n",
    "show_trace(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 学习率\n",
    "\n",
    "上述梯度下降算法中的正数$\\eta$通常叫作学习率。这是一个超参数，需要人工设定。如果使用过小的学习率，会导致$x$更新缓慢从而需要更多的迭代才能得到较好的解。\n",
    "\n",
    "下面展示使用学习率$\\eta=0.05$时自变量$x$的迭代轨迹。可见，同样迭代10次后，当学习率过小时，最终$x$的值依然与最优解存在较大偏差。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 3.4867844009999995\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\r\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 252.965625 184.15625\" width=\"252.965625pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 184.15625 \r\n",
       "L 252.965625 184.15625 \r\n",
       "L 252.965625 -0 \r\n",
       "L 0 -0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 46.965625 146.6 \r\n",
       "L 242.265625 146.6 \r\n",
       "L 242.265625 10.7 \r\n",
       "L 46.965625 10.7 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"mf6d59f7319\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.842898\" xlink:href=\"#mf6d59f7319\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- −10 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\"/>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\"/>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(45.290554 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"100.229261\" xlink:href=\"#mf6d59f7319\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- −5 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.796875 72.90625 \r\n",
       "L 49.515625 72.90625 \r\n",
       "L 49.515625 64.59375 \r\n",
       "L 19.828125 64.59375 \r\n",
       "L 19.828125 46.734375 \r\n",
       "Q 21.96875 47.46875 24.109375 47.828125 \r\n",
       "Q 26.265625 48.1875 28.421875 48.1875 \r\n",
       "Q 40.625 48.1875 47.75 41.5 \r\n",
       "Q 54.890625 34.8125 54.890625 23.390625 \r\n",
       "Q 54.890625 11.625 47.5625 5.09375 \r\n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \r\n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \r\n",
       "Q 12.796875 0.140625 7.71875 1.703125 \r\n",
       "L 7.71875 11.625 \r\n",
       "Q 12.109375 9.234375 16.796875 8.0625 \r\n",
       "Q 21.484375 6.890625 26.703125 6.890625 \r\n",
       "Q 35.15625 6.890625 40.078125 11.328125 \r\n",
       "Q 45.015625 15.765625 45.015625 23.390625 \r\n",
       "Q 45.015625 31 40.078125 35.4375 \r\n",
       "Q 35.15625 39.890625 26.703125 39.890625 \r\n",
       "Q 22.75 39.890625 18.8125 39.015625 \r\n",
       "Q 14.890625 38.140625 10.796875 36.28125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-35\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(92.858168 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"144.615625\" xlink:href=\"#mf6d59f7319\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(141.434375 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"189.001989\" xlink:href=\"#mf6d59f7319\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 5 -->\r\n",
       "      <g transform=\"translate(185.820739 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"233.388352\" xlink:href=\"#mf6d59f7319\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 10 -->\r\n",
       "      <g transform=\"translate(227.025852 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_6\">\r\n",
       "     <!-- x -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 54.890625 54.6875 \r\n",
       "L 35.109375 28.078125 \r\n",
       "L 55.90625 0 \r\n",
       "L 45.3125 0 \r\n",
       "L 29.390625 21.484375 \r\n",
       "L 13.484375 0 \r\n",
       "L 2.875 0 \r\n",
       "L 24.125 28.609375 \r\n",
       "L 4.6875 54.6875 \r\n",
       "L 15.28125 54.6875 \r\n",
       "L 29.78125 35.203125 \r\n",
       "L 44.28125 54.6875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(141.65625 174.876562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m9d6fd9c88f\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.965625\" xlink:href=\"#m9d6fd9c88f\" y=\"140.422727\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(33.603125 144.221946)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.965625\" xlink:href=\"#m9d6fd9c88f\" y=\"115.713636\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 20 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(27.240625 119.512855)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.965625\" xlink:href=\"#m9d6fd9c88f\" y=\"91.004545\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 40 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(27.240625 94.803764)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.965625\" xlink:href=\"#m9d6fd9c88f\" y=\"66.295455\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- 60 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 33.015625 40.375 \r\n",
       "Q 26.375 40.375 22.484375 35.828125 \r\n",
       "Q 18.609375 31.296875 18.609375 23.390625 \r\n",
       "Q 18.609375 15.53125 22.484375 10.953125 \r\n",
       "Q 26.375 6.390625 33.015625 6.390625 \r\n",
       "Q 39.65625 6.390625 43.53125 10.953125 \r\n",
       "Q 47.40625 15.53125 47.40625 23.390625 \r\n",
       "Q 47.40625 31.296875 43.53125 35.828125 \r\n",
       "Q 39.65625 40.375 33.015625 40.375 \r\n",
       "z\r\n",
       "M 52.59375 71.296875 \r\n",
       "L 52.59375 62.3125 \r\n",
       "Q 48.875 64.0625 45.09375 64.984375 \r\n",
       "Q 41.3125 65.921875 37.59375 65.921875 \r\n",
       "Q 27.828125 65.921875 22.671875 59.328125 \r\n",
       "Q 17.53125 52.734375 16.796875 39.40625 \r\n",
       "Q 19.671875 43.65625 24.015625 45.921875 \r\n",
       "Q 28.375 48.1875 33.59375 48.1875 \r\n",
       "Q 44.578125 48.1875 50.953125 41.515625 \r\n",
       "Q 57.328125 34.859375 57.328125 23.390625 \r\n",
       "Q 57.328125 12.15625 50.6875 5.359375 \r\n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \r\n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \r\n",
       "Q 6.984375 17.96875 6.984375 36.375 \r\n",
       "Q 6.984375 53.65625 15.1875 63.9375 \r\n",
       "Q 23.390625 74.21875 37.203125 74.21875 \r\n",
       "Q 40.921875 74.21875 44.703125 73.484375 \r\n",
       "Q 48.484375 72.75 52.59375 71.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-36\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(27.240625 70.094673)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.965625\" xlink:href=\"#m9d6fd9c88f\" y=\"41.586364\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- 80 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 34.625 \r\n",
       "Q 24.75 34.625 20.71875 30.859375 \r\n",
       "Q 16.703125 27.09375 16.703125 20.515625 \r\n",
       "Q 16.703125 13.921875 20.71875 10.15625 \r\n",
       "Q 24.75 6.390625 31.78125 6.390625 \r\n",
       "Q 38.8125 6.390625 42.859375 10.171875 \r\n",
       "Q 46.921875 13.96875 46.921875 20.515625 \r\n",
       "Q 46.921875 27.09375 42.890625 30.859375 \r\n",
       "Q 38.875 34.625 31.78125 34.625 \r\n",
       "z\r\n",
       "M 21.921875 38.8125 \r\n",
       "Q 15.578125 40.375 12.03125 44.71875 \r\n",
       "Q 8.5 49.078125 8.5 55.328125 \r\n",
       "Q 8.5 64.0625 14.71875 69.140625 \r\n",
       "Q 20.953125 74.21875 31.78125 74.21875 \r\n",
       "Q 42.671875 74.21875 48.875 69.140625 \r\n",
       "Q 55.078125 64.0625 55.078125 55.328125 \r\n",
       "Q 55.078125 49.078125 51.53125 44.71875 \r\n",
       "Q 48 40.375 41.703125 38.8125 \r\n",
       "Q 48.828125 37.15625 52.796875 32.3125 \r\n",
       "Q 56.78125 27.484375 56.78125 20.515625 \r\n",
       "Q 56.78125 9.90625 50.3125 4.234375 \r\n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.734375 -1.421875 13.25 4.234375 \r\n",
       "Q 6.78125 9.90625 6.78125 20.515625 \r\n",
       "Q 6.78125 27.484375 10.78125 32.3125 \r\n",
       "Q 14.796875 37.15625 21.921875 38.8125 \r\n",
       "z\r\n",
       "M 18.3125 54.390625 \r\n",
       "Q 18.3125 48.734375 21.84375 45.5625 \r\n",
       "Q 25.390625 42.390625 31.78125 42.390625 \r\n",
       "Q 38.140625 42.390625 41.71875 45.5625 \r\n",
       "Q 45.3125 48.734375 45.3125 54.390625 \r\n",
       "Q 45.3125 60.0625 41.71875 63.234375 \r\n",
       "Q 38.140625 66.40625 31.78125 66.40625 \r\n",
       "Q 25.390625 66.40625 21.84375 63.234375 \r\n",
       "Q 18.3125 60.0625 18.3125 54.390625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-38\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(27.240625 45.385582)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_6\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.965625\" xlink:href=\"#m9d6fd9c88f\" y=\"16.877273\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <g transform=\"translate(20.878125 20.676491)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_13\">\r\n",
       "     <!-- f(x) -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 37.109375 75.984375 \r\n",
       "L 37.109375 68.5 \r\n",
       "L 28.515625 68.5 \r\n",
       "Q 23.6875 68.5 21.796875 66.546875 \r\n",
       "Q 19.921875 64.59375 19.921875 59.515625 \r\n",
       "L 19.921875 54.6875 \r\n",
       "L 34.71875 54.6875 \r\n",
       "L 34.71875 47.703125 \r\n",
       "L 19.921875 47.703125 \r\n",
       "L 19.921875 0 \r\n",
       "L 10.890625 0 \r\n",
       "L 10.890625 47.703125 \r\n",
       "L 2.296875 47.703125 \r\n",
       "L 2.296875 54.6875 \r\n",
       "L 10.890625 54.6875 \r\n",
       "L 10.890625 58.5 \r\n",
       "Q 10.890625 67.625 15.140625 71.796875 \r\n",
       "Q 19.390625 75.984375 28.609375 75.984375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-66\"/>\r\n",
       "      <path d=\"M 31 75.875 \r\n",
       "Q 24.46875 64.65625 21.28125 53.65625 \r\n",
       "Q 18.109375 42.671875 18.109375 31.390625 \r\n",
       "Q 18.109375 20.125 21.3125 9.0625 \r\n",
       "Q 24.515625 -2 31 -13.1875 \r\n",
       "L 23.1875 -13.1875 \r\n",
       "Q 15.875 -1.703125 12.234375 9.375 \r\n",
       "Q 8.59375 20.453125 8.59375 31.390625 \r\n",
       "Q 8.59375 42.28125 12.203125 53.3125 \r\n",
       "Q 15.828125 64.359375 23.1875 75.875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-28\"/>\r\n",
       "      <path d=\"M 8.015625 75.875 \r\n",
       "L 15.828125 75.875 \r\n",
       "Q 23.140625 64.359375 26.78125 53.3125 \r\n",
       "Q 30.421875 42.28125 30.421875 31.390625 \r\n",
       "Q 30.421875 20.453125 26.78125 9.375 \r\n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \r\n",
       "L 8.015625 -13.1875 \r\n",
       "Q 14.5 -2 17.703125 9.0625 \r\n",
       "Q 20.90625 20.125 20.90625 31.390625 \r\n",
       "Q 20.90625 42.671875 17.703125 53.65625 \r\n",
       "Q 14.5 64.65625 8.015625 75.875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-29\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798438 87.271094)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\r\n",
       "      <use x=\"35.205078\" xlink:href=\"#DejaVuSans-28\"/>\r\n",
       "      <use x=\"74.21875\" xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "      <use x=\"133.398438\" xlink:href=\"#DejaVuSans-29\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_12\">\r\n",
       "    <path clip-path=\"url(#pd6262dfebb)\" d=\"M 55.842898 16.877273 \r\n",
       "L 60.281534 28.922955 \r\n",
       "L 64.72017 40.350909 \r\n",
       "L 69.158807 51.161136 \r\n",
       "L 73.597443 61.353636 \r\n",
       "L 78.03608 70.928409 \r\n",
       "L 82.474716 79.885455 \r\n",
       "L 86.025625 86.606327 \r\n",
       "L 89.576534 92.931855 \r\n",
       "L 93.127443 98.862036 \r\n",
       "L 96.678352 104.396873 \r\n",
       "L 100.229261 109.536364 \r\n",
       "L 103.78017 114.280509 \r\n",
       "L 107.33108 118.629309 \r\n",
       "L 110.881989 122.582764 \r\n",
       "L 114.432898 126.140873 \r\n",
       "L 117.983807 129.303636 \r\n",
       "L 120.646989 131.416264 \r\n",
       "L 123.31017 133.306509 \r\n",
       "L 125.973352 134.974373 \r\n",
       "L 128.636534 136.419855 \r\n",
       "L 131.299716 137.642955 \r\n",
       "L 133.962898 138.643673 \r\n",
       "L 136.62608 139.422009 \r\n",
       "L 139.289261 139.977964 \r\n",
       "L 141.952443 140.311536 \r\n",
       "L 144.615625 140.422727 \r\n",
       "L 147.278807 140.311536 \r\n",
       "L 149.941989 139.977964 \r\n",
       "L 152.60517 139.422009 \r\n",
       "L 155.268352 138.643673 \r\n",
       "L 157.931534 137.642955 \r\n",
       "L 160.594716 136.419855 \r\n",
       "L 163.257898 134.974373 \r\n",
       "L 165.92108 133.306509 \r\n",
       "L 168.584261 131.416264 \r\n",
       "L 171.247443 129.303636 \r\n",
       "L 173.910625 126.968627 \r\n",
       "L 177.461534 123.509355 \r\n",
       "L 181.012443 119.654736 \r\n",
       "L 184.563352 115.404773 \r\n",
       "L 188.114261 110.759464 \r\n",
       "L 191.66517 105.718809 \r\n",
       "L 195.21608 100.282809 \r\n",
       "L 198.766989 94.451464 \r\n",
       "L 202.317898 88.224773 \r\n",
       "L 205.868807 81.602736 \r\n",
       "L 209.419716 74.585355 \r\n",
       "L 213.858352 65.257673 \r\n",
       "L 218.296989 55.312264 \r\n",
       "L 222.735625 44.749127 \r\n",
       "L 227.174261 33.568264 \r\n",
       "L 231.612898 21.769673 \r\n",
       "L 232.500625 19.335827 \r\n",
       "L 232.500625 19.335827 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_13\">\r\n",
       "    <path clip-path=\"url(#pd6262dfebb)\" d=\"M 233.388352 16.877273 \r\n",
       "L 224.51108 40.350909 \r\n",
       "L 216.521534 59.364555 \r\n",
       "L 209.330943 74.765607 \r\n",
       "L 202.859411 87.24046 \r\n",
       "L 197.035033 97.345091 \r\n",
       "L 191.793092 105.529842 \r\n",
       "L 187.075345 112.15949 \r\n",
       "L 182.829373 117.529505 \r\n",
       "L 179.007998 121.879217 \r\n",
       "L 175.568761 125.402484 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    <defs>\r\n",
       "     <path d=\"M 0 3 \r\n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \r\n",
       "C 2.683901 1.55874 3 0.795609 3 0 \r\n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \r\n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \r\n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \r\n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \r\n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \r\n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \r\n",
       "z\r\n",
       "\" id=\"m68955d756f\" style=\"stroke:#ff7f0e;\"/>\r\n",
       "    </defs>\r\n",
       "    <g clip-path=\"url(#pd6262dfebb)\">\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"233.388352\" xlink:href=\"#m68955d756f\" y=\"16.877273\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"224.51108\" xlink:href=\"#m68955d756f\" y=\"40.350909\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"216.521534\" xlink:href=\"#m68955d756f\" y=\"59.364555\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"209.330943\" xlink:href=\"#m68955d756f\" y=\"74.765607\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"202.859411\" xlink:href=\"#m68955d756f\" y=\"87.24046\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"197.035033\" xlink:href=\"#m68955d756f\" y=\"97.345091\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"191.793092\" xlink:href=\"#m68955d756f\" y=\"105.529842\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"187.075345\" xlink:href=\"#m68955d756f\" y=\"112.15949\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"182.829373\" xlink:href=\"#m68955d756f\" y=\"117.529505\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"179.007998\" xlink:href=\"#m68955d756f\" y=\"121.879217\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"175.568761\" xlink:href=\"#m68955d756f\" y=\"125.402484\"/>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 46.965625 146.6 \r\n",
       "L 46.965625 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 242.265625 146.6 \r\n",
       "L 242.265625 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 46.965625 146.6 \r\n",
       "L 242.265625 146.6 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 46.965625 10.7 \r\n",
       "L 242.265625 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"pd6262dfebb\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"46.965625\" y=\"10.7\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_trace(gd(0.05))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果使用过大的学习率，$\\left|\\eta f'(x)\\right|$可能会过大从而使前面提到的一阶泰勒展开公式不再成立：这时我们无法保证迭代$x$会降低$f(x)$的值。\n",
    "\n",
    "举个例子，当设学习率$\\eta=1.1$时，可以看到$x$不断越过（overshoot）最优解$x=0$并逐渐发散。\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 61.917364224000096\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\r\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 259.328125 184.15625\" width=\"259.328125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 184.15625 \r\n",
       "L 259.328125 184.15625 \r\n",
       "L 259.328125 -0 \r\n",
       "L 0 -0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 53.328125 146.6 \r\n",
       "L 248.628125 146.6 \r\n",
       "L 248.628125 10.7 \r\n",
       "L 53.328125 10.7 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m7b97179bfb\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"79.29167\" xlink:href=\"#m7b97179bfb\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- −50 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\"/>\r\n",
       "       <path d=\"M 10.796875 72.90625 \r\n",
       "L 49.515625 72.90625 \r\n",
       "L 49.515625 64.59375 \r\n",
       "L 19.828125 64.59375 \r\n",
       "L 19.828125 46.734375 \r\n",
       "Q 21.96875 47.46875 24.109375 47.828125 \r\n",
       "Q 26.265625 48.1875 28.421875 48.1875 \r\n",
       "Q 40.625 48.1875 47.75 41.5 \r\n",
       "Q 54.890625 34.8125 54.890625 23.390625 \r\n",
       "Q 54.890625 11.625 47.5625 5.09375 \r\n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \r\n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \r\n",
       "Q 12.796875 0.140625 7.71875 1.703125 \r\n",
       "L 7.71875 11.625 \r\n",
       "Q 12.109375 9.234375 16.796875 8.0625 \r\n",
       "Q 21.484375 6.890625 26.703125 6.890625 \r\n",
       "Q 35.15625 6.890625 40.078125 11.328125 \r\n",
       "Q 45.015625 15.765625 45.015625 23.390625 \r\n",
       "Q 45.015625 31 40.078125 35.4375 \r\n",
       "Q 35.15625 39.890625 26.703125 39.890625 \r\n",
       "Q 22.75 39.890625 18.8125 39.015625 \r\n",
       "Q 14.890625 38.140625 10.796875 36.28125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-35\"/>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(68.739326 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"115.134897\" xlink:href=\"#m7b97179bfb\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- −25 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(104.582554 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"150.978125\" xlink:href=\"#m7b97179bfb\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(147.796875 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"186.821353\" xlink:href=\"#m7b97179bfb\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 25 -->\r\n",
       "      <g transform=\"translate(180.458853 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"222.66458\" xlink:href=\"#m7b97179bfb\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 50 -->\r\n",
       "      <g transform=\"translate(216.30208 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_6\">\r\n",
       "     <!-- x -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 54.890625 54.6875 \r\n",
       "L 35.109375 28.078125 \r\n",
       "L 55.90625 0 \r\n",
       "L 45.3125 0 \r\n",
       "L 29.390625 21.484375 \r\n",
       "L 13.484375 0 \r\n",
       "L 2.875 0 \r\n",
       "L 24.125 28.609375 \r\n",
       "L 4.6875 54.6875 \r\n",
       "L 15.28125 54.6875 \r\n",
       "L 29.78125 35.203125 \r\n",
       "L 44.28125 54.6875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(148.01875 174.876562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"mcbfa0cc0b4\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"53.328125\" xlink:href=\"#mcbfa0cc0b4\" y=\"140.422737\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(39.965625 144.221956)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"53.328125\" xlink:href=\"#mcbfa0cc0b4\" y=\"108.197072\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 1000 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 111.996291)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"190.869141\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"53.328125\" xlink:href=\"#mcbfa0cc0b4\" y=\"75.971407\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 2000 -->\r\n",
       "      <g transform=\"translate(20.878125 79.770626)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"190.869141\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"53.328125\" xlink:href=\"#mcbfa0cc0b4\" y=\"43.745743\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- 3000 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 40.578125 39.3125 \r\n",
       "Q 47.65625 37.796875 51.625 33 \r\n",
       "Q 55.609375 28.21875 55.609375 21.1875 \r\n",
       "Q 55.609375 10.40625 48.1875 4.484375 \r\n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \r\n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \r\n",
       "Q 12.796875 0.390625 7.625 2.203125 \r\n",
       "L 7.625 11.71875 \r\n",
       "Q 11.71875 9.328125 16.59375 8.109375 \r\n",
       "Q 21.484375 6.890625 26.8125 6.890625 \r\n",
       "Q 36.078125 6.890625 40.9375 10.546875 \r\n",
       "Q 45.796875 14.203125 45.796875 21.1875 \r\n",
       "Q 45.796875 27.640625 41.28125 31.265625 \r\n",
       "Q 36.765625 34.90625 28.71875 34.90625 \r\n",
       "L 20.21875 34.90625 \r\n",
       "L 20.21875 43.015625 \r\n",
       "L 29.109375 43.015625 \r\n",
       "Q 36.375 43.015625 40.234375 45.921875 \r\n",
       "Q 44.09375 48.828125 44.09375 54.296875 \r\n",
       "Q 44.09375 59.90625 40.109375 62.90625 \r\n",
       "Q 36.140625 65.921875 28.71875 65.921875 \r\n",
       "Q 24.65625 65.921875 20.015625 65.03125 \r\n",
       "Q 15.375 64.15625 9.8125 62.3125 \r\n",
       "L 9.8125 71.09375 \r\n",
       "Q 15.4375 72.65625 20.34375 73.4375 \r\n",
       "Q 25.25 74.21875 29.59375 74.21875 \r\n",
       "Q 40.828125 74.21875 47.359375 69.109375 \r\n",
       "Q 53.90625 64.015625 53.90625 55.328125 \r\n",
       "Q 53.90625 49.265625 50.4375 45.09375 \r\n",
       "Q 46.96875 40.921875 40.578125 39.3125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-33\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 47.544961)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"190.869141\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"53.328125\" xlink:href=\"#mcbfa0cc0b4\" y=\"11.520078\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- 4000 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 15.319297)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"190.869141\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_12\">\r\n",
       "     <!-- f(x) -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 37.109375 75.984375 \r\n",
       "L 37.109375 68.5 \r\n",
       "L 28.515625 68.5 \r\n",
       "Q 23.6875 68.5 21.796875 66.546875 \r\n",
       "Q 19.921875 64.59375 19.921875 59.515625 \r\n",
       "L 19.921875 54.6875 \r\n",
       "L 34.71875 54.6875 \r\n",
       "L 34.71875 47.703125 \r\n",
       "L 19.921875 47.703125 \r\n",
       "L 19.921875 0 \r\n",
       "L 10.890625 0 \r\n",
       "L 10.890625 47.703125 \r\n",
       "L 2.296875 47.703125 \r\n",
       "L 2.296875 54.6875 \r\n",
       "L 10.890625 54.6875 \r\n",
       "L 10.890625 58.5 \r\n",
       "Q 10.890625 67.625 15.140625 71.796875 \r\n",
       "Q 19.390625 75.984375 28.609375 75.984375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-66\"/>\r\n",
       "      <path d=\"M 31 75.875 \r\n",
       "Q 24.46875 64.65625 21.28125 53.65625 \r\n",
       "Q 18.109375 42.671875 18.109375 31.390625 \r\n",
       "Q 18.109375 20.125 21.3125 9.0625 \r\n",
       "Q 24.515625 -2 31 -13.1875 \r\n",
       "L 23.1875 -13.1875 \r\n",
       "Q 15.875 -1.703125 12.234375 9.375 \r\n",
       "Q 8.59375 20.453125 8.59375 31.390625 \r\n",
       "Q 8.59375 42.28125 12.203125 53.3125 \r\n",
       "Q 15.828125 64.359375 23.1875 75.875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-28\"/>\r\n",
       "      <path d=\"M 8.015625 75.875 \r\n",
       "L 15.828125 75.875 \r\n",
       "Q 23.140625 64.359375 26.78125 53.3125 \r\n",
       "Q 30.421875 42.28125 30.421875 31.390625 \r\n",
       "Q 30.421875 20.453125 26.78125 9.375 \r\n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \r\n",
       "L 8.015625 -13.1875 \r\n",
       "Q 14.5 -2 17.703125 9.0625 \r\n",
       "Q 20.90625 20.125 20.90625 31.390625 \r\n",
       "Q 20.90625 42.671875 17.703125 53.65625 \r\n",
       "Q 14.5 64.65625 8.015625 75.875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-29\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798437 87.271094)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\r\n",
       "      <use x=\"35.205078\" xlink:href=\"#DejaVuSans-28\"/>\r\n",
       "      <use x=\"74.21875\" xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "      <use x=\"133.398438\" xlink:href=\"#DejaVuSans-29\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_11\">\r\n",
       "    <path clip-path=\"url(#p03679d48d1)\" d=\"M 62.205398 16.877273 \r\n",
       "L 66.793331 29.317383 \r\n",
       "L 71.237891 40.739372 \r\n",
       "L 75.539078 51.20318 \r\n",
       "L 79.840266 61.086926 \r\n",
       "L 83.99808 70.089833 \r\n",
       "L 88.012522 78.267973 \r\n",
       "L 92.026963 85.940815 \r\n",
       "L 95.898032 92.861076 \r\n",
       "L 99.625727 99.080961 \r\n",
       "L 103.353423 104.865155 \r\n",
       "L 106.937746 110.016003 \r\n",
       "L 110.378696 114.581843 \r\n",
       "L 113.819646 118.776443 \r\n",
       "L 117.117222 122.447909 \r\n",
       "L 120.271427 125.640712 \r\n",
       "L 123.425631 128.521571 \r\n",
       "L 126.436462 130.980483 \r\n",
       "L 129.447293 133.155165 \r\n",
       "L 132.314751 134.962041 \r\n",
       "L 135.182209 136.511111 \r\n",
       "L 138.049667 137.802376 \r\n",
       "L 140.773753 138.790286 \r\n",
       "L 143.497838 139.545526 \r\n",
       "L 146.221923 140.068097 \r\n",
       "L 148.946009 140.357998 \r\n",
       "L 151.670094 140.41523 \r\n",
       "L 154.394179 140.239793 \r\n",
       "L 157.118265 139.831687 \r\n",
       "L 159.84235 139.190911 \r\n",
       "L 162.566435 138.317467 \r\n",
       "L 165.29052 137.211352 \r\n",
       "L 168.014606 135.872569 \r\n",
       "L 170.882064 134.211963 \r\n",
       "L 173.749522 132.293551 \r\n",
       "L 176.760353 130.001757 \r\n",
       "L 179.771184 127.425731 \r\n",
       "L 182.925388 124.422183 \r\n",
       "L 186.079593 121.106691 \r\n",
       "L 189.377169 117.306959 \r\n",
       "L 192.674746 113.166279 \r\n",
       "L 196.115696 108.482065 \r\n",
       "L 199.556646 103.42661 \r\n",
       "L 203.140969 97.765747 \r\n",
       "L 206.868665 91.451138 \r\n",
       "L 210.59636 84.700837 \r\n",
       "L 214.467429 77.229761 \r\n",
       "L 218.48187 68.985702 \r\n",
       "L 222.496312 60.236345 \r\n",
       "L 226.654126 50.641821 \r\n",
       "L 230.955314 40.146057 \r\n",
       "L 235.399874 28.691047 \r\n",
       "L 239.701061 17.015823 \r\n",
       "L 239.701061 17.015823 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_12\">\r\n",
       "    <path clip-path=\"url(#p03679d48d1)\" d=\"M 165.315416 137.200171 \r\n",
       "L 133.773376 135.782241 \r\n",
       "L 171.623824 133.740423 \r\n",
       "L 126.203286 130.800205 \r\n",
       "L 180.707932 126.566291 \r\n",
       "L 115.302357 120.469455 \r\n",
       "L 193.789047 111.690011 \r\n",
       "L 99.605019 99.047611 \r\n",
       "L 212.625852 80.842556 \r\n",
       "L 77.000852 54.627276 \r\n",
       "L 239.750852 16.877273 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    <defs>\r\n",
       "     <path d=\"M 0 3 \r\n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \r\n",
       "C 2.683901 1.55874 3 0.795609 3 0 \r\n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \r\n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \r\n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \r\n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \r\n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \r\n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \r\n",
       "z\r\n",
       "\" id=\"m8836bccb68\" style=\"stroke:#ff7f0e;\"/>\r\n",
       "    </defs>\r\n",
       "    <g clip-path=\"url(#p03679d48d1)\">\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"165.315416\" xlink:href=\"#m8836bccb68\" y=\"137.200171\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"133.773376\" xlink:href=\"#m8836bccb68\" y=\"135.782241\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"171.623824\" xlink:href=\"#m8836bccb68\" y=\"133.740423\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"126.203286\" xlink:href=\"#m8836bccb68\" y=\"130.800205\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"180.707932\" xlink:href=\"#m8836bccb68\" y=\"126.566291\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"115.302357\" xlink:href=\"#m8836bccb68\" y=\"120.469455\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"193.789047\" xlink:href=\"#m8836bccb68\" y=\"111.690011\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"99.605019\" xlink:href=\"#m8836bccb68\" y=\"99.047611\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"212.625852\" xlink:href=\"#m8836bccb68\" y=\"80.842556\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"77.000852\" xlink:href=\"#m8836bccb68\" y=\"54.627276\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"239.750852\" xlink:href=\"#m8836bccb68\" y=\"16.877273\"/>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 53.328125 146.6 \r\n",
       "L 53.328125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 248.628125 146.6 \r\n",
       "L 248.628125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 53.328125 146.6 \r\n",
       "L 248.628125 146.6 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 53.328125 10.7 \r\n",
       "L 248.628125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p03679d48d1\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"53.328125\" y=\"10.7\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_trace(gd(1.1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 多维梯度下降\n",
    "\n",
    "在了解了一维梯度下降之后，我们再考虑一种更广义的情况：目标函数的输入为向量，输出为标量。假设目标函数$f: \\mathbb{R}^d \\rightarrow \\mathbb{R}$的输入是一个$d$维向量$\\boldsymbol{x} = [x_1, x_2, \\ldots, x_d]^\\top$。目标函数$f(\\boldsymbol{x})$有关$\\boldsymbol{x}$的梯度是一个由$d$个偏导数组成的向量：\n",
    "\n",
    "$$\\nabla_{\\boldsymbol{x}} f(\\boldsymbol{x}) = \\bigg[\\frac{\\partial f(\\boldsymbol{x})}{\\partial x_1}, \\frac{\\partial f(\\boldsymbol{x})}{\\partial x_2}, \\ldots, \\frac{\\partial f(\\boldsymbol{x})}{\\partial x_d}\\bigg]^\\top.$$\n",
    "\n",
    "为表示简洁，我们用$\\nabla f(\\boldsymbol{x})$代替$\\nabla_{\\boldsymbol{x}} f(\\boldsymbol{x})$。梯度中每个偏导数元素$\\partial f(\\boldsymbol{x})/\\partial x_i$代表着$f$在$\\boldsymbol{x}$有关输入$x_i$的变化率。为了测量$f$沿着单位向量$\\boldsymbol{u}$（即$\\|\\boldsymbol{u}\\|=1$）方向上的变化率，在多元微积分中，我们定义$f$在$\\boldsymbol{x}$上沿着$\\boldsymbol{u}$方向的方向导数为\n",
    "\n",
    "$$\\text{D}_{\\boldsymbol{u}} f(\\boldsymbol{x}) = \\lim_{h \\rightarrow 0}  \\frac{f(\\boldsymbol{x} + h \\boldsymbol{u}) - f(\\boldsymbol{x})}{h}.$$\n",
    "\n",
    "依据方向导数性质[1，14.6节定理三]，以上方向导数可以改写为\n",
    "\n",
    "$$\\text{D}_{\\boldsymbol{u}} f(\\boldsymbol{x}) = \\nabla f(\\boldsymbol{x}) \\cdot \\boldsymbol{u}.$$\n",
    "\n",
    "方向导数$\\text{D}_{\\boldsymbol{u}} f(\\boldsymbol{x})$给出了$f$在$\\boldsymbol{x}$上沿着所有可能方向的变化率。为了最小化$f$，我们希望找到$f$能被降低最快的方向。因此，我们可以通过单位向量$\\boldsymbol{u}$来最小化方向导数$\\text{D}_{\\boldsymbol{u}} f(\\boldsymbol{x})$。\n",
    "\n",
    "由于$\\text{D}_{\\boldsymbol{u}} f(\\boldsymbol{x}) = \\|\\nabla f(\\boldsymbol{x})\\| \\cdot \\|\\boldsymbol{u}\\|  \\cdot \\text{cos} (\\theta) = \\|\\nabla f(\\boldsymbol{x})\\|  \\cdot \\text{cos} (\\theta)$，\n",
    "其中$\\theta$为梯度$\\nabla f(\\boldsymbol{x})$和单位向量$\\boldsymbol{u}$之间的夹角，当$\\theta = \\pi$时，$\\text{cos}(\\theta)$取得最小值$-1$。因此，当$\\boldsymbol{u}$在梯度方向$\\nabla f(\\boldsymbol{x})$的相反方向时，方向导数$\\text{D}_{\\boldsymbol{u}} f(\\boldsymbol{x})$被最小化。因此，我们可能通过梯度下降算法来不断降低目标函数$f$的值：\n",
    "\n",
    "$$\\boldsymbol{x} \\leftarrow \\boldsymbol{x} - \\eta \\nabla f(\\boldsymbol{x}).$$\n",
    "\n",
    "同样，其中$\\eta$（取正数）称作学习率。\n",
    "\n",
    "下面我们构造一个输入为二维向量$\\boldsymbol{x} = [x_1, x_2]^\\top$和输出为标量的目标函数$f(\\boldsymbol{x})=x_1^2+2x_2^2$。那么，梯度$\\nabla f(\\boldsymbol{x}) = [2x_1, 4x_2]^\\top$。我们将观察梯度下降从初始位置$[-5,-2]$开始对自变量$\\boldsymbol{x}$的迭代轨迹。我们先定义两个辅助函数，第一个函数使用给定的自变量更新函数，从初始位置$[-5,-2]$开始迭代自变量$\\boldsymbol{x}$共20次，第二个函数对自变量$\\boldsymbol{x}$的迭代轨迹进行可视化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "def train_2d(trainer):  # 本函数将保存在d2lzh_pytorch包中方便以后使用\n",
    "    x1, x2, s1, s2 = -5, -2, 0, 0  # s1和s2是自变量状态，本章后续几节会使用\n",
    "    results = [(x1, x2)]\n",
    "    for i in range(20):\n",
    "        x1, x2, s1, s2 = trainer(x1, x2, s1, s2)\n",
    "        results.append((x1, x2))\n",
    "    print('epoch %d, x1 %f, x2 %f' % (i + 1, x1, x2))\n",
    "    return results\n",
    "\n",
    "def show_trace_2d(f, results):  # 本函数将保存在d2lzh_pytorch包中方便以后使用\n",
    "    plt.plot(*zip(*results), '-o', color='#ff7f0e')\n",
    "    x1, x2 = np.meshgrid(np.arange(-5.5, 1.0, 0.1), np.arange(-3.0, 1.0, 0.1))\n",
    "    plt.contour(x1, x2, f(x1, x2), colors='#1f77b4')\n",
    "    plt.xlabel('x1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 20, x1 -0.057646, x2 -0.000073\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\r\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 234.942188 184.15625\" width=\"234.942188pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 184.15625 \r\n",
       "L 234.942188 184.15625 \r\n",
       "L 234.942188 -0 \r\n",
       "L 0 -0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 28.942188 146.6 \r\n",
       "L 224.242188 146.6 \r\n",
       "L 224.242188 10.7 \r\n",
       "L 28.942188 10.7 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m32eb83f838\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"74.715625\" xlink:href=\"#m32eb83f838\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- −4 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\"/>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(67.344531 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"135.746875\" xlink:href=\"#m32eb83f838\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- −2 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(128.375781 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"196.778125\" xlink:href=\"#m32eb83f838\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(193.596875 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_4\">\r\n",
       "     <!-- x1 -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 54.890625 54.6875 \r\n",
       "L 35.109375 28.078125 \r\n",
       "L 55.90625 0 \r\n",
       "L 45.3125 0 \r\n",
       "L 29.390625 21.484375 \r\n",
       "L 13.484375 0 \r\n",
       "L 2.875 0 \r\n",
       "L 24.125 28.609375 \r\n",
       "L 4.6875 54.6875 \r\n",
       "L 15.28125 54.6875 \r\n",
       "L 29.78125 35.203125 \r\n",
       "L 44.28125 54.6875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\"/>\r\n",
       "      <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(120.451563 174.876562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "      <use x=\"59.179688\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m0098e07365\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"28.942188\" xlink:href=\"#m0098e07365\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- −3 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 40.578125 39.3125 \r\n",
       "Q 47.65625 37.796875 51.625 33 \r\n",
       "Q 55.609375 28.21875 55.609375 21.1875 \r\n",
       "Q 55.609375 10.40625 48.1875 4.484375 \r\n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \r\n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \r\n",
       "Q 12.796875 0.390625 7.625 2.203125 \r\n",
       "L 7.625 11.71875 \r\n",
       "Q 11.71875 9.328125 16.59375 8.109375 \r\n",
       "Q 21.484375 6.890625 26.8125 6.890625 \r\n",
       "Q 36.078125 6.890625 40.9375 10.546875 \r\n",
       "Q 45.796875 14.203125 45.796875 21.1875 \r\n",
       "Q 45.796875 27.640625 41.28125 31.265625 \r\n",
       "Q 36.765625 34.90625 28.71875 34.90625 \r\n",
       "L 20.21875 34.90625 \r\n",
       "L 20.21875 43.015625 \r\n",
       "L 29.109375 43.015625 \r\n",
       "Q 36.375 43.015625 40.234375 45.921875 \r\n",
       "Q 44.09375 48.828125 44.09375 54.296875 \r\n",
       "Q 44.09375 59.90625 40.109375 62.90625 \r\n",
       "Q 36.140625 65.921875 28.71875 65.921875 \r\n",
       "Q 24.65625 65.921875 20.015625 65.03125 \r\n",
       "Q 15.375 64.15625 9.8125 62.3125 \r\n",
       "L 9.8125 71.09375 \r\n",
       "Q 15.4375 72.65625 20.34375 73.4375 \r\n",
       "Q 25.25 74.21875 29.59375 74.21875 \r\n",
       "Q 40.828125 74.21875 47.359375 69.109375 \r\n",
       "Q 53.90625 64.015625 53.90625 55.328125 \r\n",
       "Q 53.90625 49.265625 50.4375 45.09375 \r\n",
       "Q 46.96875 40.921875 40.578125 39.3125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-33\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(7.2 150.399219)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-33\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"28.942188\" xlink:href=\"#m0098e07365\" y=\"111.753846\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- −2 -->\r\n",
       "      <g transform=\"translate(7.2 115.553065)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"28.942188\" xlink:href=\"#m0098e07365\" y=\"76.907692\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- −1 -->\r\n",
       "      <g transform=\"translate(7.2 80.706911)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"28.942188\" xlink:href=\"#m0098e07365\" y=\"42.061538\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(15.579688 45.860757)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"LineCollection_1\">\r\n",
       "    <path clip-path=\"url(#p37e247880d)\" d=\"M 224.242188 108.127963 \r\n",
       "L 223.703676 108.269231 \r\n",
       "L 221.190625 108.894675 \r\n",
       "L 218.139063 109.564793 \r\n",
       "L 215.0875 110.145562 \r\n",
       "L 212.035938 110.636982 \r\n",
       "L 208.984375 111.039053 \r\n",
       "L 205.932813 111.351775 \r\n",
       "L 202.88125 111.575148 \r\n",
       "L 199.829688 111.709172 \r\n",
       "L 196.778125 111.753846 \r\n",
       "L 196.778125 111.753846 \r\n",
       "L 196.778125 111.753846 \r\n",
       "L 193.726563 111.709172 \r\n",
       "L 190.675 111.575148 \r\n",
       "L 187.623438 111.351775 \r\n",
       "L 184.571875 111.039053 \r\n",
       "L 181.520313 110.636982 \r\n",
       "L 178.46875 110.145562 \r\n",
       "L 175.417188 109.564793 \r\n",
       "L 172.365625 108.894675 \r\n",
       "L 169.852574 108.269231 \r\n",
       "L 169.314063 108.127963 \r\n",
       "L 166.2625 107.233264 \r\n",
       "L 163.210938 106.244387 \r\n",
       "L 160.159375 105.161331 \r\n",
       "L 159.182875 104.784615 \r\n",
       "L 157.107813 103.938352 \r\n",
       "L 154.05625 102.594286 \r\n",
       "L 151.320366 101.3 \r\n",
       "L 151.004688 101.141608 \r\n",
       "L 147.953125 99.504895 \r\n",
       "L 144.994034 97.815385 \r\n",
       "L 144.901563 97.759181 \r\n",
       "L 141.85 95.79206 \r\n",
       "L 139.705659 94.330769 \r\n",
       "L 138.798438 93.669894 \r\n",
       "L 135.746875 91.32679 \r\n",
       "L 135.151448 90.846154 \r\n",
       "L 132.695312 88.716667 \r\n",
       "L 131.205015 87.361538 \r\n",
       "L 129.64375 85.828308 \r\n",
       "L 127.745 83.876923 \r\n",
       "L 126.592188 82.58913 \r\n",
       "L 124.709309 80.392308 \r\n",
       "L 123.540625 78.898901 \r\n",
       "L 122.045982 76.907692 \r\n",
       "L 120.489063 74.615182 \r\n",
       "L 119.711213 73.423077 \r\n",
       "L 117.676838 69.938462 \r\n",
       "L 117.4375 69.473846 \r\n",
       "L 115.940507 66.453846 \r\n",
       "L 114.443514 62.969231 \r\n",
       "L 114.385938 62.810839 \r\n",
       "L 113.220795 59.484615 \r\n",
       "L 112.222102 56 \r\n",
       "L 111.445341 52.515385 \r\n",
       "L 111.334375 51.818462 \r\n",
       "L 110.906086 49.030769 \r\n",
       "L 110.584868 45.546154 \r\n",
       "L 110.477796 42.061538 \r\n",
       "L 110.584868 38.576923 \r\n",
       "L 110.906086 35.092308 \r\n",
       "L 111.334375 32.304615 \r\n",
       "L 111.445341 31.607692 \r\n",
       "L 112.222102 28.123077 \r\n",
       "L 113.220795 24.638462 \r\n",
       "L 114.385938 21.312238 \r\n",
       "L 114.443514 21.153846 \r\n",
       "L 115.940507 17.669231 \r\n",
       "L 117.4375 14.649231 \r\n",
       "L 117.676838 14.184615 \r\n",
       "L 119.711213 10.7 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"LineCollection_2\">\r\n",
       "    <path clip-path=\"url(#p37e247880d)\" d=\"M 224.242188 138.078531 \r\n",
       "L 221.190625 138.617063 \r\n",
       "L 218.139063 139.092238 \r\n",
       "L 215.0875 139.504056 \r\n",
       "L 213.977841 139.630769 \r\n",
       "L 212.035938 139.844737 \r\n",
       "L 208.984375 140.119838 \r\n",
       "L 205.932813 140.333806 \r\n",
       "L 202.88125 140.48664 \r\n",
       "L 199.829688 140.57834 \r\n",
       "L 196.778125 140.608907 \r\n",
       "L 193.726563 140.57834 \r\n",
       "L 190.675 140.48664 \r\n",
       "L 187.623438 140.333806 \r\n",
       "L 184.571875 140.119838 \r\n",
       "L 181.520313 139.844737 \r\n",
       "L 179.578409 139.630769 \r\n",
       "L 178.46875 139.504056 \r\n",
       "L 175.417188 139.092238 \r\n",
       "L 172.365625 138.617063 \r\n",
       "L 169.314063 138.078531 \r\n",
       "L 166.2625 137.476643 \r\n",
       "L 163.210938 136.811399 \r\n",
       "L 160.424728 136.146154 \r\n",
       "L 160.159375 136.080406 \r\n",
       "L 157.107813 135.258563 \r\n",
       "L 154.05625 134.370972 \r\n",
       "L 151.004688 133.417634 \r\n",
       "L 148.740625 132.661538 \r\n",
       "L 147.953125 132.388235 \r\n",
       "L 144.901563 131.26086 \r\n",
       "L 141.85 130.065158 \r\n",
       "L 139.705659 129.176923 \r\n",
       "L 138.798438 128.785793 \r\n",
       "L 135.746875 127.399058 \r\n",
       "L 132.695312 125.941209 \r\n",
       "L 132.198547 125.692308 \r\n",
       "L 129.64375 124.357774 \r\n",
       "L 126.592188 122.689607 \r\n",
       "L 125.748138 122.207692 \r\n",
       "L 123.540625 120.891282 \r\n",
       "L 120.489063 118.994103 \r\n",
       "L 120.070221 118.723077 \r\n",
       "L 117.4375 116.94025 \r\n",
       "L 115.019281 115.238462 \r\n",
       "L 114.385938 114.771013 \r\n",
       "L 111.334375 112.433771 \r\n",
       "L 110.477796 111.753846 \r\n",
       "L 108.282813 109.922189 \r\n",
       "L 106.369121 108.269231 \r\n",
       "L 105.23125 107.233264 \r\n",
       "L 102.629918 104.784615 \r\n",
       "L 102.179688 104.336593 \r\n",
       "L 99.225 101.3 \r\n",
       "L 99.128125 101.194406 \r\n",
       "L 96.12351 97.815385 \r\n",
       "L 96.076563 97.759181 \r\n",
       "L 93.298274 94.330769 \r\n",
       "L 93.025 93.970292 \r\n",
       "L 90.725272 90.846154 \r\n",
       "L 89.973438 89.749145 \r\n",
       "L 88.383187 87.361538 \r\n",
       "L 86.921875 84.992 \r\n",
       "L 86.253039 83.876923 \r\n",
       "L 84.330137 80.392308 \r\n",
       "L 83.870312 79.47967 \r\n",
       "L 82.609 76.907692 \r\n",
       "L 81.062875 73.423077 \r\n",
       "L 80.81875 72.808145 \r\n",
       "L 79.709091 69.938462 \r\n",
       "L 78.52017 66.453846 \r\n",
       "L 77.767188 63.907396 \r\n",
       "L 77.496796 62.969231 \r\n",
       "L 76.646994 59.484615 \r\n",
       "L 75.951701 56 \r\n",
       "L 75.410918 52.515385 \r\n",
       "L 75.024644 49.030769 \r\n",
       "L 74.79288 45.546154 \r\n",
       "L 74.715625 42.061538 \r\n",
       "L 74.79288 38.576923 \r\n",
       "L 75.024644 35.092308 \r\n",
       "L 75.410918 31.607692 \r\n",
       "L 75.951701 28.123077 \r\n",
       "L 76.646994 24.638462 \r\n",
       "L 77.496796 21.153846 \r\n",
       "L 77.767188 20.21568 \r\n",
       "L 78.52017 17.669231 \r\n",
       "L 79.709091 14.184615 \r\n",
       "L 80.81875 11.314932 \r\n",
       "L 81.062875 10.7 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"LineCollection_3\">\r\n",
       "    <path clip-path=\"url(#p37e247880d)\" d=\"M 122.045982 146.6 \r\n",
       "L 120.489063 145.861734 \r\n",
       "L 117.4375 144.355671 \r\n",
       "L 115.019281 143.115385 \r\n",
       "L 114.385938 142.77915 \r\n",
       "L 111.334375 141.097976 \r\n",
       "L 108.764638 139.630769 \r\n",
       "L 108.282813 139.345664 \r\n",
       "L 105.23125 137.476643 \r\n",
       "L 103.130174 136.146154 \r\n",
       "L 102.179688 135.521553 \r\n",
       "L 99.128125 133.450508 \r\n",
       "L 98.001394 132.661538 \r\n",
       "L 96.076563 131.26086 \r\n",
       "L 93.298274 129.176923 \r\n",
       "L 93.025 128.963579 \r\n",
       "L 89.973438 126.510126 \r\n",
       "L 88.984903 125.692308 \r\n",
       "L 86.921875 123.91293 \r\n",
       "L 84.998973 122.207692 \r\n",
       "L 83.870313 121.162308 \r\n",
       "L 81.307 118.723077 \r\n",
       "L 80.81875 118.236852 \r\n",
       "L 77.88608 115.238462 \r\n",
       "L 77.767188 115.110976 \r\n",
       "L 74.715625 111.753846 \r\n",
       "L 74.715625 111.753846 \r\n",
       "L 71.777083 108.269231 \r\n",
       "L 71.664063 108.127963 \r\n",
       "L 69.05369 104.784615 \r\n",
       "L 68.6125 104.187253 \r\n",
       "L 66.530257 101.3 \r\n",
       "L 65.560938 99.874476 \r\n",
       "L 64.192996 97.815385 \r\n",
       "L 62.509375 95.117618 \r\n",
       "L 62.029354 94.330769 \r\n",
       "L 60.040695 90.846154 \r\n",
       "L 59.457813 89.749145 \r\n",
       "L 58.217067 87.361538 \r\n",
       "L 56.540385 83.876923 \r\n",
       "L 56.40625 83.573913 \r\n",
       "L 55.028125 80.392308 \r\n",
       "L 53.65 76.907692 \r\n",
       "L 53.354688 76.082389 \r\n",
       "L 52.423158 73.423077 \r\n",
       "L 51.33102 69.938462 \r\n",
       "L 50.367368 66.453846 \r\n",
       "L 50.303125 66.185799 \r\n",
       "L 49.548099 62.969231 \r\n",
       "L 48.855992 59.484615 \r\n",
       "L 48.289723 56 \r\n",
       "L 47.849291 52.515385 \r\n",
       "L 47.534697 49.030769 \r\n",
       "L 47.345941 45.546154 \r\n",
       "L 47.283022 42.061538 \r\n",
       "L 47.345941 38.576923 \r\n",
       "L 47.534697 35.092308 \r\n",
       "L 47.849291 31.607692 \r\n",
       "L 48.289723 28.123077 \r\n",
       "L 48.855992 24.638462 \r\n",
       "L 49.548099 21.153846 \r\n",
       "L 50.303125 17.937278 \r\n",
       "L 50.367368 17.669231 \r\n",
       "L 51.33102 14.184615 \r\n",
       "L 52.423158 10.7 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"LineCollection_4\">\r\n",
       "    <path clip-path=\"url(#p37e247880d)\" d=\"M 82.609 146.6 \r\n",
       "L 80.81875 145.300652 \r\n",
       "L 77.88608 143.115385 \r\n",
       "L 77.767188 143.023684 \r\n",
       "L 74.715625 140.608907 \r\n",
       "L 73.510069 139.630769 \r\n",
       "L 71.664063 138.078531 \r\n",
       "L 69.421348 136.146154 \r\n",
       "L 68.6125 135.422932 \r\n",
       "L 65.596838 132.661538 \r\n",
       "L 65.560938 132.627376 \r\n",
       "L 62.509375 129.655204 \r\n",
       "L 62.029354 129.176923 \r\n",
       "L 59.457813 126.510126 \r\n",
       "L 58.686538 125.692308 \r\n",
       "L 56.40625 123.171522 \r\n",
       "L 55.553125 122.207692 \r\n",
       "L 53.354688 119.61359 \r\n",
       "L 52.615888 118.723077 \r\n",
       "L 50.303125 115.805725 \r\n",
       "L 49.862693 115.238462 \r\n",
       "L 47.283022 111.753846 \r\n",
       "L 47.251562 111.709172 \r\n",
       "L 44.878125 108.269231 \r\n",
       "L 44.2 107.233264 \r\n",
       "L 42.628899 104.784615 \r\n",
       "L 41.148438 102.345385 \r\n",
       "L 40.526274 101.3 \r\n",
       "L 38.570904 97.815385 \r\n",
       "L 38.096875 96.916129 \r\n",
       "L 36.76 94.330769 \r\n",
       "L 35.074375 90.846154 \r\n",
       "L 35.045312 90.781624 \r\n",
       "L 33.533791 87.361538 \r\n",
       "L 32.107827 83.876923 \r\n",
       "L 31.99375 83.573913 \r\n",
       "L 30.817919 80.392308 \r\n",
       "L 29.642087 76.907692 \r\n",
       "L 28.942188 74.615182 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"LineCollection_5\">\r\n",
       "    <path clip-path=\"url(#p37e247880d)\" d=\"M 53.65 146.6 \r\n",
       "L 53.354688 146.334224 \r\n",
       "L 50.303125 143.528814 \r\n",
       "L 49.862693 143.115385 \r\n",
       "L 47.251563 140.57834 \r\n",
       "L 46.296023 139.630769 \r\n",
       "L 44.2 137.476643 \r\n",
       "L 42.931033 136.146154 \r\n",
       "L 41.148438 134.206604 \r\n",
       "L 39.755977 132.661538 \r\n",
       "L 38.096875 130.748416 \r\n",
       "L 36.76 129.176923 \r\n",
       "L 35.045312 127.079042 \r\n",
       "L 33.933061 125.692308 \r\n",
       "L 31.99375 123.171522 \r\n",
       "L 31.265854 122.207692 \r\n",
       "L 28.942188 118.994103 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"LineCollection_6\">\r\n",
       "    <path clip-path=\"url(#p37e247880d)\" d=\"M 29.642087 146.6 \r\n",
       "L 28.942188 145.861734 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_8\">\r\n",
       "    <path clip-path=\"url(#p37e247880d)\" d=\"M 44.2 111.753846 \r\n",
       "L 74.715625 83.876923 \r\n",
       "L 99.128125 67.150769 \r\n",
       "L 118.658125 57.115077 \r\n",
       "L 134.282125 51.093662 \r\n",
       "L 146.781325 47.480812 \r\n",
       "L 156.780685 45.313103 \r\n",
       "L 164.780173 44.012477 \r\n",
       "L 171.179763 43.232102 \r\n",
       "L 176.299436 42.763876 \r\n",
       "L 180.395174 42.482941 \r\n",
       "L 183.671764 42.31438 \r\n",
       "L 186.293036 42.213243 \r\n",
       "L 188.390054 42.152561 \r\n",
       "L 190.067668 42.116152 \r\n",
       "L 191.409759 42.094307 \r\n",
       "L 192.483433 42.081199 \r\n",
       "L 193.342371 42.073335 \r\n",
       "L 194.029522 42.068616 \r\n",
       "L 194.579242 42.065785 \r\n",
       "L 195.019019 42.064087 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    <defs>\r\n",
       "     <path d=\"M 0 3 \r\n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \r\n",
       "C 2.683901 1.55874 3 0.795609 3 0 \r\n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \r\n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \r\n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \r\n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \r\n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \r\n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \r\n",
       "z\r\n",
       "\" id=\"m3ba2074297\" style=\"stroke:#ff7f0e;\"/>\r\n",
       "    </defs>\r\n",
       "    <g clip-path=\"url(#p37e247880d)\">\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"44.2\" xlink:href=\"#m3ba2074297\" y=\"111.753846\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"74.715625\" xlink:href=\"#m3ba2074297\" y=\"83.876923\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"99.128125\" xlink:href=\"#m3ba2074297\" y=\"67.150769\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"118.658125\" xlink:href=\"#m3ba2074297\" y=\"57.115077\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"134.282125\" xlink:href=\"#m3ba2074297\" y=\"51.093662\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"146.781325\" xlink:href=\"#m3ba2074297\" y=\"47.480812\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"156.780685\" xlink:href=\"#m3ba2074297\" y=\"45.313103\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"164.780173\" xlink:href=\"#m3ba2074297\" y=\"44.012477\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"171.179763\" xlink:href=\"#m3ba2074297\" y=\"43.232102\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"176.299436\" xlink:href=\"#m3ba2074297\" y=\"42.763876\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"180.395174\" xlink:href=\"#m3ba2074297\" y=\"42.482941\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"183.671764\" xlink:href=\"#m3ba2074297\" y=\"42.31438\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"186.293036\" xlink:href=\"#m3ba2074297\" y=\"42.213243\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"188.390054\" xlink:href=\"#m3ba2074297\" y=\"42.152561\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"190.067668\" xlink:href=\"#m3ba2074297\" y=\"42.116152\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"191.409759\" xlink:href=\"#m3ba2074297\" y=\"42.094307\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"192.483433\" xlink:href=\"#m3ba2074297\" y=\"42.081199\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"193.342371\" xlink:href=\"#m3ba2074297\" y=\"42.073335\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"194.029522\" xlink:href=\"#m3ba2074297\" y=\"42.068616\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"194.579242\" xlink:href=\"#m3ba2074297\" y=\"42.065785\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"195.019019\" xlink:href=\"#m3ba2074297\" y=\"42.064087\"/>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 28.942188 146.6 \r\n",
       "L 28.942188 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 224.242188 146.6 \r\n",
       "L 224.242188 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 28.942188 146.6 \r\n",
       "L 224.242188 146.6 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 28.942188 10.7 \r\n",
       "L 224.242188 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p37e247880d\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"28.942188\" y=\"10.7\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "eta = 0.1\n",
    "\n",
    "def f_2d(x1, x2):  # 目标函数\n",
    "    return x1 ** 2 + 2 * x2 ** 2\n",
    "\n",
    "def gd_2d(x1, x2, s1, s2):\n",
    "    return (x1 - eta * 2 * x1, x2 - eta * 4 * x2, 0, 0)\n",
    "\n",
    "show_trace_2d(f_2d, train_2d(gd_2d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 随机梯度下降\n",
    "\n",
    "在深度学习里，目标函数通常是训练数据集中有关各个样本的损失函数的平均。设$f_i(\\boldsymbol{x})$是有关索引为$i$的训练数据样本的损失函数，$n$是训练数据样本数，$\\boldsymbol{x}$是模型的参数向量，那么目标函数定义为\n",
    "\n",
    "$$f(\\boldsymbol{x}) = \\frac{1}{n} \\sum_{i = 1}^n f_i(\\boldsymbol{x}).$$\n",
    "\n",
    "目标函数在$\\boldsymbol{x}$处的梯度计算为\n",
    "\n",
    "$$\\nabla f(\\boldsymbol{x}) = \\frac{1}{n} \\sum_{i = 1}^n \\nabla f_i(\\boldsymbol{x}).$$\n",
    "\n",
    "如果使用梯度下降，每次自变量迭代的计算开销为$\\mathcal{O}(n)$，它随着$n$线性增长。因此，当训练数据样本数很大时，梯度下降每次迭代的计算开销很高。\n",
    "\n",
    "**随机梯度下降（stochastic gradient descent，SGD）减少了每次迭代的计算开销。在随机梯度下降的每次迭代中，我们随机均匀采样的一个样本索引$i\\in\\{1,\\ldots,n\\}$，并计算梯度$\\nabla f_i(\\boldsymbol{x})$来迭代$\\boldsymbol{x}$：**\n",
    "\n",
    "$$\\boldsymbol{x} \\leftarrow \\boldsymbol{x} - \\eta \\nabla f_i(\\boldsymbol{x}).$$\n",
    "\n",
    "这里$\\eta$同样是学习率。可以看到每次迭代的计算开销从梯度下降的$\\mathcal{O}(n)$降到了常数$\\mathcal{O}(1)$。值得强调的是，随机梯度$\\nabla f_i(\\boldsymbol{x})$是对梯度$\\nabla f(\\boldsymbol{x})$的无偏估计：\n",
    "\n",
    "$$E_i \\nabla f_i(\\boldsymbol{x}) = \\frac{1}{n} \\sum_{i = 1}^n \\nabla f_i(\\boldsymbol{x}) = \\nabla f(\\boldsymbol{x}).$$\n",
    "\n",
    "这意味着，平均来说，随机梯度是对梯度的一个良好的估计。\n",
    "\n",
    "下面我们通过在梯度中添加均值为0的随机噪声来模拟随机梯度下降，以此来比较它与梯度下降的区别。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 20, x1 -0.019040, x2 -0.119766\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\r\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 234.942188 184.15625\" width=\"234.942188pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 184.15625 \r\n",
       "L 234.942188 184.15625 \r\n",
       "L 234.942188 -0 \r\n",
       "L 0 -0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 28.942188 146.6 \r\n",
       "L 224.242188 146.6 \r\n",
       "L 224.242188 10.7 \r\n",
       "L 28.942188 10.7 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"mdd22e983b1\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"74.715625\" xlink:href=\"#mdd22e983b1\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- −4 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.59375 35.5 \r\n",
       "L 73.1875 35.5 \r\n",
       "L 73.1875 27.203125 \r\n",
       "L 10.59375 27.203125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\"/>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(67.344531 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"135.746875\" xlink:href=\"#mdd22e983b1\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- −2 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(128.375781 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"196.778125\" xlink:href=\"#mdd22e983b1\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(193.596875 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_4\">\r\n",
       "     <!-- x1 -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 54.890625 54.6875 \r\n",
       "L 35.109375 28.078125 \r\n",
       "L 55.90625 0 \r\n",
       "L 45.3125 0 \r\n",
       "L 29.390625 21.484375 \r\n",
       "L 13.484375 0 \r\n",
       "L 2.875 0 \r\n",
       "L 24.125 28.609375 \r\n",
       "L 4.6875 54.6875 \r\n",
       "L 15.28125 54.6875 \r\n",
       "L 29.78125 35.203125 \r\n",
       "L 44.28125 54.6875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\"/>\r\n",
       "      <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(120.451563 174.876562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "      <use x=\"59.179688\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m4632bc9638\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"28.942188\" xlink:href=\"#m4632bc9638\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- −3 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 40.578125 39.3125 \r\n",
       "Q 47.65625 37.796875 51.625 33 \r\n",
       "Q 55.609375 28.21875 55.609375 21.1875 \r\n",
       "Q 55.609375 10.40625 48.1875 4.484375 \r\n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \r\n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \r\n",
       "Q 12.796875 0.390625 7.625 2.203125 \r\n",
       "L 7.625 11.71875 \r\n",
       "Q 11.71875 9.328125 16.59375 8.109375 \r\n",
       "Q 21.484375 6.890625 26.8125 6.890625 \r\n",
       "Q 36.078125 6.890625 40.9375 10.546875 \r\n",
       "Q 45.796875 14.203125 45.796875 21.1875 \r\n",
       "Q 45.796875 27.640625 41.28125 31.265625 \r\n",
       "Q 36.765625 34.90625 28.71875 34.90625 \r\n",
       "L 20.21875 34.90625 \r\n",
       "L 20.21875 43.015625 \r\n",
       "L 29.109375 43.015625 \r\n",
       "Q 36.375 43.015625 40.234375 45.921875 \r\n",
       "Q 44.09375 48.828125 44.09375 54.296875 \r\n",
       "Q 44.09375 59.90625 40.109375 62.90625 \r\n",
       "Q 36.140625 65.921875 28.71875 65.921875 \r\n",
       "Q 24.65625 65.921875 20.015625 65.03125 \r\n",
       "Q 15.375 64.15625 9.8125 62.3125 \r\n",
       "L 9.8125 71.09375 \r\n",
       "Q 15.4375 72.65625 20.34375 73.4375 \r\n",
       "Q 25.25 74.21875 29.59375 74.21875 \r\n",
       "Q 40.828125 74.21875 47.359375 69.109375 \r\n",
       "Q 53.90625 64.015625 53.90625 55.328125 \r\n",
       "Q 53.90625 49.265625 50.4375 45.09375 \r\n",
       "Q 46.96875 40.921875 40.578125 39.3125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-33\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(7.2 150.399219)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-33\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"28.942188\" xlink:href=\"#m4632bc9638\" y=\"111.753846\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- −2 -->\r\n",
       "      <g transform=\"translate(7.2 115.553065)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"28.942188\" xlink:href=\"#m4632bc9638\" y=\"76.907692\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- −1 -->\r\n",
       "      <g transform=\"translate(7.2 80.706911)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"28.942188\" xlink:href=\"#m4632bc9638\" y=\"42.061538\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(15.579688 45.860757)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"LineCollection_1\">\r\n",
       "    <path clip-path=\"url(#pd3b85b2ec6)\" d=\"M 224.242188 108.127963 \r\n",
       "L 223.703676 108.269231 \r\n",
       "L 221.190625 108.894675 \r\n",
       "L 218.139063 109.564793 \r\n",
       "L 215.0875 110.145562 \r\n",
       "L 212.035938 110.636982 \r\n",
       "L 208.984375 111.039053 \r\n",
       "L 205.932813 111.351775 \r\n",
       "L 202.88125 111.575148 \r\n",
       "L 199.829688 111.709172 \r\n",
       "L 196.778125 111.753846 \r\n",
       "L 196.778125 111.753846 \r\n",
       "L 196.778125 111.753846 \r\n",
       "L 193.726563 111.709172 \r\n",
       "L 190.675 111.575148 \r\n",
       "L 187.623438 111.351775 \r\n",
       "L 184.571875 111.039053 \r\n",
       "L 181.520313 110.636982 \r\n",
       "L 178.46875 110.145562 \r\n",
       "L 175.417188 109.564793 \r\n",
       "L 172.365625 108.894675 \r\n",
       "L 169.852574 108.269231 \r\n",
       "L 169.314063 108.127963 \r\n",
       "L 166.2625 107.233264 \r\n",
       "L 163.210938 106.244387 \r\n",
       "L 160.159375 105.161331 \r\n",
       "L 159.182875 104.784615 \r\n",
       "L 157.107813 103.938352 \r\n",
       "L 154.05625 102.594286 \r\n",
       "L 151.320366 101.3 \r\n",
       "L 151.004688 101.141608 \r\n",
       "L 147.953125 99.504895 \r\n",
       "L 144.994034 97.815385 \r\n",
       "L 144.901563 97.759181 \r\n",
       "L 141.85 95.79206 \r\n",
       "L 139.705659 94.330769 \r\n",
       "L 138.798438 93.669894 \r\n",
       "L 135.746875 91.32679 \r\n",
       "L 135.151448 90.846154 \r\n",
       "L 132.695312 88.716667 \r\n",
       "L 131.205015 87.361538 \r\n",
       "L 129.64375 85.828308 \r\n",
       "L 127.745 83.876923 \r\n",
       "L 126.592188 82.58913 \r\n",
       "L 124.709309 80.392308 \r\n",
       "L 123.540625 78.898901 \r\n",
       "L 122.045982 76.907692 \r\n",
       "L 120.489063 74.615182 \r\n",
       "L 119.711213 73.423077 \r\n",
       "L 117.676838 69.938462 \r\n",
       "L 117.4375 69.473846 \r\n",
       "L 115.940507 66.453846 \r\n",
       "L 114.443514 62.969231 \r\n",
       "L 114.385938 62.810839 \r\n",
       "L 113.220795 59.484615 \r\n",
       "L 112.222102 56 \r\n",
       "L 111.445341 52.515385 \r\n",
       "L 111.334375 51.818462 \r\n",
       "L 110.906086 49.030769 \r\n",
       "L 110.584868 45.546154 \r\n",
       "L 110.477796 42.061538 \r\n",
       "L 110.584868 38.576923 \r\n",
       "L 110.906086 35.092308 \r\n",
       "L 111.334375 32.304615 \r\n",
       "L 111.445341 31.607692 \r\n",
       "L 112.222102 28.123077 \r\n",
       "L 113.220795 24.638462 \r\n",
       "L 114.385938 21.312238 \r\n",
       "L 114.443514 21.153846 \r\n",
       "L 115.940507 17.669231 \r\n",
       "L 117.4375 14.649231 \r\n",
       "L 117.676838 14.184615 \r\n",
       "L 119.711213 10.7 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"LineCollection_2\">\r\n",
       "    <path clip-path=\"url(#pd3b85b2ec6)\" d=\"M 224.242188 138.078531 \r\n",
       "L 221.190625 138.617063 \r\n",
       "L 218.139063 139.092238 \r\n",
       "L 215.0875 139.504056 \r\n",
       "L 213.977841 139.630769 \r\n",
       "L 212.035938 139.844737 \r\n",
       "L 208.984375 140.119838 \r\n",
       "L 205.932813 140.333806 \r\n",
       "L 202.88125 140.48664 \r\n",
       "L 199.829688 140.57834 \r\n",
       "L 196.778125 140.608907 \r\n",
       "L 193.726563 140.57834 \r\n",
       "L 190.675 140.48664 \r\n",
       "L 187.623438 140.333806 \r\n",
       "L 184.571875 140.119838 \r\n",
       "L 181.520313 139.844737 \r\n",
       "L 179.578409 139.630769 \r\n",
       "L 178.46875 139.504056 \r\n",
       "L 175.417188 139.092238 \r\n",
       "L 172.365625 138.617063 \r\n",
       "L 169.314063 138.078531 \r\n",
       "L 166.2625 137.476643 \r\n",
       "L 163.210938 136.811399 \r\n",
       "L 160.424728 136.146154 \r\n",
       "L 160.159375 136.080406 \r\n",
       "L 157.107813 135.258563 \r\n",
       "L 154.05625 134.370972 \r\n",
       "L 151.004688 133.417634 \r\n",
       "L 148.740625 132.661538 \r\n",
       "L 147.953125 132.388235 \r\n",
       "L 144.901563 131.26086 \r\n",
       "L 141.85 130.065158 \r\n",
       "L 139.705659 129.176923 \r\n",
       "L 138.798438 128.785793 \r\n",
       "L 135.746875 127.399058 \r\n",
       "L 132.695312 125.941209 \r\n",
       "L 132.198547 125.692308 \r\n",
       "L 129.64375 124.357774 \r\n",
       "L 126.592188 122.689607 \r\n",
       "L 125.748138 122.207692 \r\n",
       "L 123.540625 120.891282 \r\n",
       "L 120.489063 118.994103 \r\n",
       "L 120.070221 118.723077 \r\n",
       "L 117.4375 116.94025 \r\n",
       "L 115.019281 115.238462 \r\n",
       "L 114.385938 114.771013 \r\n",
       "L 111.334375 112.433771 \r\n",
       "L 110.477796 111.753846 \r\n",
       "L 108.282813 109.922189 \r\n",
       "L 106.369121 108.269231 \r\n",
       "L 105.23125 107.233264 \r\n",
       "L 102.629918 104.784615 \r\n",
       "L 102.179688 104.336593 \r\n",
       "L 99.225 101.3 \r\n",
       "L 99.128125 101.194406 \r\n",
       "L 96.12351 97.815385 \r\n",
       "L 96.076563 97.759181 \r\n",
       "L 93.298274 94.330769 \r\n",
       "L 93.025 93.970292 \r\n",
       "L 90.725272 90.846154 \r\n",
       "L 89.973438 89.749145 \r\n",
       "L 88.383187 87.361538 \r\n",
       "L 86.921875 84.992 \r\n",
       "L 86.253039 83.876923 \r\n",
       "L 84.330137 80.392308 \r\n",
       "L 83.870312 79.47967 \r\n",
       "L 82.609 76.907692 \r\n",
       "L 81.062875 73.423077 \r\n",
       "L 80.81875 72.808145 \r\n",
       "L 79.709091 69.938462 \r\n",
       "L 78.52017 66.453846 \r\n",
       "L 77.767188 63.907396 \r\n",
       "L 77.496796 62.969231 \r\n",
       "L 76.646994 59.484615 \r\n",
       "L 75.951701 56 \r\n",
       "L 75.410918 52.515385 \r\n",
       "L 75.024644 49.030769 \r\n",
       "L 74.79288 45.546154 \r\n",
       "L 74.715625 42.061538 \r\n",
       "L 74.79288 38.576923 \r\n",
       "L 75.024644 35.092308 \r\n",
       "L 75.410918 31.607692 \r\n",
       "L 75.951701 28.123077 \r\n",
       "L 76.646994 24.638462 \r\n",
       "L 77.496796 21.153846 \r\n",
       "L 77.767188 20.21568 \r\n",
       "L 78.52017 17.669231 \r\n",
       "L 79.709091 14.184615 \r\n",
       "L 80.81875 11.314932 \r\n",
       "L 81.062875 10.7 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"LineCollection_3\">\r\n",
       "    <path clip-path=\"url(#pd3b85b2ec6)\" d=\"M 122.045982 146.6 \r\n",
       "L 120.489063 145.861734 \r\n",
       "L 117.4375 144.355671 \r\n",
       "L 115.019281 143.115385 \r\n",
       "L 114.385938 142.77915 \r\n",
       "L 111.334375 141.097976 \r\n",
       "L 108.764638 139.630769 \r\n",
       "L 108.282813 139.345664 \r\n",
       "L 105.23125 137.476643 \r\n",
       "L 103.130174 136.146154 \r\n",
       "L 102.179688 135.521553 \r\n",
       "L 99.128125 133.450508 \r\n",
       "L 98.001394 132.661538 \r\n",
       "L 96.076563 131.26086 \r\n",
       "L 93.298274 129.176923 \r\n",
       "L 93.025 128.963579 \r\n",
       "L 89.973438 126.510126 \r\n",
       "L 88.984903 125.692308 \r\n",
       "L 86.921875 123.91293 \r\n",
       "L 84.998973 122.207692 \r\n",
       "L 83.870313 121.162308 \r\n",
       "L 81.307 118.723077 \r\n",
       "L 80.81875 118.236852 \r\n",
       "L 77.88608 115.238462 \r\n",
       "L 77.767188 115.110976 \r\n",
       "L 74.715625 111.753846 \r\n",
       "L 74.715625 111.753846 \r\n",
       "L 71.777083 108.269231 \r\n",
       "L 71.664063 108.127963 \r\n",
       "L 69.05369 104.784615 \r\n",
       "L 68.6125 104.187253 \r\n",
       "L 66.530257 101.3 \r\n",
       "L 65.560938 99.874476 \r\n",
       "L 64.192996 97.815385 \r\n",
       "L 62.509375 95.117618 \r\n",
       "L 62.029354 94.330769 \r\n",
       "L 60.040695 90.846154 \r\n",
       "L 59.457813 89.749145 \r\n",
       "L 58.217067 87.361538 \r\n",
       "L 56.540385 83.876923 \r\n",
       "L 56.40625 83.573913 \r\n",
       "L 55.028125 80.392308 \r\n",
       "L 53.65 76.907692 \r\n",
       "L 53.354688 76.082389 \r\n",
       "L 52.423158 73.423077 \r\n",
       "L 51.33102 69.938462 \r\n",
       "L 50.367368 66.453846 \r\n",
       "L 50.303125 66.185799 \r\n",
       "L 49.548099 62.969231 \r\n",
       "L 48.855992 59.484615 \r\n",
       "L 48.289723 56 \r\n",
       "L 47.849291 52.515385 \r\n",
       "L 47.534697 49.030769 \r\n",
       "L 47.345941 45.546154 \r\n",
       "L 47.283022 42.061538 \r\n",
       "L 47.345941 38.576923 \r\n",
       "L 47.534697 35.092308 \r\n",
       "L 47.849291 31.607692 \r\n",
       "L 48.289723 28.123077 \r\n",
       "L 48.855992 24.638462 \r\n",
       "L 49.548099 21.153846 \r\n",
       "L 50.303125 17.937278 \r\n",
       "L 50.367368 17.669231 \r\n",
       "L 51.33102 14.184615 \r\n",
       "L 52.423158 10.7 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"LineCollection_4\">\r\n",
       "    <path clip-path=\"url(#pd3b85b2ec6)\" d=\"M 82.609 146.6 \r\n",
       "L 80.81875 145.300652 \r\n",
       "L 77.88608 143.115385 \r\n",
       "L 77.767188 143.023684 \r\n",
       "L 74.715625 140.608907 \r\n",
       "L 73.510069 139.630769 \r\n",
       "L 71.664063 138.078531 \r\n",
       "L 69.421348 136.146154 \r\n",
       "L 68.6125 135.422932 \r\n",
       "L 65.596838 132.661538 \r\n",
       "L 65.560938 132.627376 \r\n",
       "L 62.509375 129.655204 \r\n",
       "L 62.029354 129.176923 \r\n",
       "L 59.457813 126.510126 \r\n",
       "L 58.686538 125.692308 \r\n",
       "L 56.40625 123.171522 \r\n",
       "L 55.553125 122.207692 \r\n",
       "L 53.354688 119.61359 \r\n",
       "L 52.615888 118.723077 \r\n",
       "L 50.303125 115.805725 \r\n",
       "L 49.862693 115.238462 \r\n",
       "L 47.283022 111.753846 \r\n",
       "L 47.251562 111.709172 \r\n",
       "L 44.878125 108.269231 \r\n",
       "L 44.2 107.233264 \r\n",
       "L 42.628899 104.784615 \r\n",
       "L 41.148438 102.345385 \r\n",
       "L 40.526274 101.3 \r\n",
       "L 38.570904 97.815385 \r\n",
       "L 38.096875 96.916129 \r\n",
       "L 36.76 94.330769 \r\n",
       "L 35.074375 90.846154 \r\n",
       "L 35.045312 90.781624 \r\n",
       "L 33.533791 87.361538 \r\n",
       "L 32.107827 83.876923 \r\n",
       "L 31.99375 83.573913 \r\n",
       "L 30.817919 80.392308 \r\n",
       "L 29.642087 76.907692 \r\n",
       "L 28.942188 74.615182 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"LineCollection_5\">\r\n",
       "    <path clip-path=\"url(#pd3b85b2ec6)\" d=\"M 53.65 146.6 \r\n",
       "L 53.354688 146.334224 \r\n",
       "L 50.303125 143.528814 \r\n",
       "L 49.862693 143.115385 \r\n",
       "L 47.251563 140.57834 \r\n",
       "L 46.296023 139.630769 \r\n",
       "L 44.2 137.476643 \r\n",
       "L 42.931033 136.146154 \r\n",
       "L 41.148438 134.206604 \r\n",
       "L 39.755977 132.661538 \r\n",
       "L 38.096875 130.748416 \r\n",
       "L 36.76 129.176923 \r\n",
       "L 35.045312 127.079042 \r\n",
       "L 33.933061 125.692308 \r\n",
       "L 31.99375 123.171522 \r\n",
       "L 31.265854 122.207692 \r\n",
       "L 28.942188 118.994103 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"LineCollection_6\">\r\n",
       "    <path clip-path=\"url(#pd3b85b2ec6)\" d=\"M 29.642087 146.6 \r\n",
       "L 28.942188 145.861734 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_8\">\r\n",
       "    <path clip-path=\"url(#pd3b85b2ec6)\" d=\"M 44.2 111.753846 \r\n",
       "L 75.266131 85.146943 \r\n",
       "L 102.36033 66.51041 \r\n",
       "L 121.173367 58.413753 \r\n",
       "L 137.683719 60.137853 \r\n",
       "L 145.101199 51.343606 \r\n",
       "L 155.033186 50.45382 \r\n",
       "L 166.147006 48.147159 \r\n",
       "L 170.966942 47.865362 \r\n",
       "L 170.301474 36.19407 \r\n",
       "L 173.700055 35.292768 \r\n",
       "L 179.671777 40.423134 \r\n",
       "L 180.746457 41.655781 \r\n",
       "L 181.110805 45.207476 \r\n",
       "L 182.491544 42.296075 \r\n",
       "L 191.668663 34.970252 \r\n",
       "L 187.579175 39.908899 \r\n",
       "L 192.367586 42.808751 \r\n",
       "L 187.189681 42.086537 \r\n",
       "L 192.405959 46.034752 \r\n",
       "L 196.197107 46.234925 \r\n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    <defs>\r\n",
       "     <path d=\"M 0 3 \r\n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \r\n",
       "C 2.683901 1.55874 3 0.795609 3 0 \r\n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \r\n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \r\n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \r\n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \r\n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \r\n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \r\n",
       "z\r\n",
       "\" id=\"m73094af6c8\" style=\"stroke:#ff7f0e;\"/>\r\n",
       "    </defs>\r\n",
       "    <g clip-path=\"url(#pd3b85b2ec6)\">\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"44.2\" xlink:href=\"#m73094af6c8\" y=\"111.753846\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"75.266131\" xlink:href=\"#m73094af6c8\" y=\"85.146943\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"102.36033\" xlink:href=\"#m73094af6c8\" y=\"66.51041\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"121.173367\" xlink:href=\"#m73094af6c8\" y=\"58.413753\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"137.683719\" xlink:href=\"#m73094af6c8\" y=\"60.137853\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"145.101199\" xlink:href=\"#m73094af6c8\" y=\"51.343606\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"155.033186\" xlink:href=\"#m73094af6c8\" y=\"50.45382\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"166.147006\" xlink:href=\"#m73094af6c8\" y=\"48.147159\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"170.966942\" xlink:href=\"#m73094af6c8\" y=\"47.865362\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"170.301474\" xlink:href=\"#m73094af6c8\" y=\"36.19407\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"173.700055\" xlink:href=\"#m73094af6c8\" y=\"35.292768\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"179.671777\" xlink:href=\"#m73094af6c8\" y=\"40.423134\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"180.746457\" xlink:href=\"#m73094af6c8\" y=\"41.655781\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"181.110805\" xlink:href=\"#m73094af6c8\" y=\"45.207476\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"182.491544\" xlink:href=\"#m73094af6c8\" y=\"42.296075\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"191.668663\" xlink:href=\"#m73094af6c8\" y=\"34.970252\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"187.579175\" xlink:href=\"#m73094af6c8\" y=\"39.908899\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"192.367586\" xlink:href=\"#m73094af6c8\" y=\"42.808751\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"187.189681\" xlink:href=\"#m73094af6c8\" y=\"42.086537\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"192.405959\" xlink:href=\"#m73094af6c8\" y=\"46.034752\"/>\r\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"196.197107\" xlink:href=\"#m73094af6c8\" y=\"46.234925\"/>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 28.942188 146.6 \r\n",
       "L 28.942188 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 224.242188 146.6 \r\n",
       "L 224.242188 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 28.942188 146.6 \r\n",
       "L 224.242188 146.6 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 28.942188 10.7 \r\n",
       "L 224.242188 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"pd3b85b2ec6\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"28.942188\" y=\"10.7\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def sgd_2d(x1, x2, s1, s2):\n",
    "    return (x1 - eta * (2 * x1 + np.random.normal(0.1)),\n",
    "            x2 - eta * (4 * x2 + np.random.normal(0.1)), 0, 0)\n",
    "\n",
    "show_trace_2d(f_2d, train_2d(sgd_2d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，随机梯度下降中自变量的迭代轨迹相对于梯度下降中的来说更为曲折。这是由于实验所添加的噪声使模拟的随机梯度的准确度下降。在实际中，这些噪声通常指训练数据集中的无意义的干扰。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 小结\n",
    "\n",
    "- 使用适当的学习率，沿着梯度反方向更新自变量可能降低目标函数值。梯度下降重复这一更新过程直到得到满足要求的解。\n",
    "- 学习率过大或过小都有问题。一个合适的学习率通常是需要通过多次实验找到的。\n",
    "- 当训练数据集的样本较多时，梯度下降每次迭代的计算开销较大，因而随机梯度下降通常更受青睐。\n",
    "\n",
    "## 参考文献\n",
    "\n",
    "[1] Stewart, J. (2010). Calculus: early transcendentals. 7th ed. Cengage Learning.\n",
    "\n",
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# mini -batch SGD 小批量随机梯度下降\n",
    "\n",
    "1. 梯度下降使用整个训练数据集来计算梯度，因此它有时也被称为批量梯度下降（batch gradient descent）\n",
    "2. 随机梯度下降 SGD 在每次迭代中只随机采样一个样本来计算梯度。\n",
    "3. 我们还可以在每轮迭代中随机均匀采样多个样本来组成一个小批量，然后使用这个小批量来计算梯度。 mini-batch gradient descent\n",
    "\n",
    "下面就来描述小批量随机梯度下降。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设目标函数$f(\\boldsymbol{x}): \\mathbb{R}^d \\rightarrow \\mathbb{R}$。在迭代开始前的时间步设为0。该时间步的自变量记为$\\boldsymbol{x}_0\\in \\mathbb{R}^d$，通常由随机初始化得到。在接下来的每一个时间步$t>0$中，小批量随机梯度下降随机均匀采样一个由训练数据样本索引组成的小批量$\\mathcal{B}_t$。我们可以通过重复采样（sampling with replacement）或者不重复采样（sampling without replacement）得到一个小批量中的各个样本。前者允许同一个小批量中出现重复的样本，后者则不允许如此，且更常见。对于这两者间的任一种方式，都可以使用\n",
    "$$\n",
    "\\boldsymbol{g}_t \\leftarrow \\nabla f_{\\mathcal{B}_t}(\\boldsymbol{x}_{t-1}) = \\frac{1}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}_t}\\nabla f_i(\\boldsymbol{x}_{t-1})\n",
    "$$\n",
    "来计算时间步$t$的小批量$\\mathcal{B}_t$上目标函数位于$\\boldsymbol{x}_{t-1}$处的梯度$\\boldsymbol{g}_t$。这里$|\\mathcal{B}|$代表批量大小，即小批量中样本的个数，是一个超参数。同随机梯度一样，重复采样所得的小批量随机梯度$\\boldsymbol{g}_t$也是对梯度$\\nabla f(\\boldsymbol{x}_{t-1})$的无偏估计。给定学习率$\\eta_t$（取正数），小批量随机梯度下降对自变量的迭代如下：\n",
    "\n",
    "$$\\boldsymbol{x}_t \\leftarrow \\boldsymbol{x}_{t-1} - \\eta_t \\boldsymbol{g}_t.$$\n",
    "\n",
    "基于随机采样得到的梯度的方差在迭代过程中无法减小，因此在实际中，（小批量）随机梯度下降的学习率可以在迭代过程中自我衰减，例如$\\eta_t=\\eta t^\\alpha$（通常$\\alpha=-1$或者$-0.5$）、$\\eta_t = \\eta \\alpha^t$（如$\\alpha=0.95$）或者每迭代若干次后将学习率衰减一次。如此一来，学习率和（小批量）随机梯度乘积的方差会减小。而梯度下降在迭代过程中一直使用目标函数的真实梯度，无须自我衰减学习率。\n",
    "\n",
    "小批量随机梯度下降中每次迭代的计算开销为$\\mathcal{O}(|\\mathcal{B}|)$。当批量大小为1时，该算法即为随机梯度下降；当批量大小等于训练数据样本数时，该算法即为梯度下降。当批量较小时，每次迭代中使用的样本少，这会导致并行处理和内存使用效率变低。这使得在计算同样数目样本的情况下比使用更大批量时所花时间更多。当批量较大时，每个小批量梯度里可能含有更多的冗余信息。为了得到较好的解，批量较大时比批量较小时需要计算的样本数目可能更多，例如增大迭代周期数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import time\n",
    "import torch\n",
    "from torch import nn, optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1500, 5])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_data_ch7():  # 本函数已保存在d2lzh_pytorch包中方便以后使用\n",
    "    data = np.genfromtxt('../Data/airfoil_self_noise.dat', delimiter='\\t')\n",
    "    data = (data - data.mean(axis=0)) / data.std(axis=0)\n",
    "    return torch.tensor(data[:1500, :-1], dtype=torch.float32), \\\n",
    "    torch.tensor(data[:1500, -1], dtype=torch.float32) # 前1500个样本(每个样本5个特征)\n",
    "\n",
    "features, labels = get_data_ch7()\n",
    "features.shape # torch.Size([1500, 5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 0.245300, 0.063829 sec per epoch\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\r\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 249.78125 184.15625\" width=\"249.78125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 184.15625 \r\n",
       "L 249.78125 184.15625 \r\n",
       "L 249.78125 -0 \r\n",
       "L 0 -0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 43.78125 146.6 \r\n",
       "L 239.08125 146.6 \r\n",
       "L 239.08125 10.7 \r\n",
       "L 43.78125 10.7 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m138fd34d07\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.658523\" xlink:href=\"#m138fd34d07\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 0.0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\"/>\r\n",
       "       <path d=\"M 10.6875 12.40625 \r\n",
       "L 21 12.40625 \r\n",
       "L 21 0 \r\n",
       "L 10.6875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2e\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(44.70696 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"97.044886\" xlink:href=\"#m138fd34d07\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 0.5 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.796875 72.90625 \r\n",
       "L 49.515625 72.90625 \r\n",
       "L 49.515625 64.59375 \r\n",
       "L 19.828125 64.59375 \r\n",
       "L 19.828125 46.734375 \r\n",
       "Q 21.96875 47.46875 24.109375 47.828125 \r\n",
       "Q 26.265625 48.1875 28.421875 48.1875 \r\n",
       "Q 40.625 48.1875 47.75 41.5 \r\n",
       "Q 54.890625 34.8125 54.890625 23.390625 \r\n",
       "Q 54.890625 11.625 47.5625 5.09375 \r\n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \r\n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \r\n",
       "Q 12.796875 0.140625 7.71875 1.703125 \r\n",
       "L 7.71875 11.625 \r\n",
       "Q 12.109375 9.234375 16.796875 8.0625 \r\n",
       "Q 21.484375 6.890625 26.703125 6.890625 \r\n",
       "Q 35.15625 6.890625 40.078125 11.328125 \r\n",
       "Q 45.015625 15.765625 45.015625 23.390625 \r\n",
       "Q 45.015625 31 40.078125 35.4375 \r\n",
       "Q 35.15625 39.890625 26.703125 39.890625 \r\n",
       "Q 22.75 39.890625 18.8125 39.015625 \r\n",
       "Q 14.890625 38.140625 10.796875 36.28125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-35\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(89.093324 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"141.43125\" xlink:href=\"#m138fd34d07\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 1.0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(133.479688 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"185.817614\" xlink:href=\"#m138fd34d07\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 1.5 -->\r\n",
       "      <g transform=\"translate(177.866051 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"230.203977\" xlink:href=\"#m138fd34d07\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 2.0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(222.252415 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_6\">\r\n",
       "     <!-- epoch -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 56.203125 29.59375 \r\n",
       "L 56.203125 25.203125 \r\n",
       "L 14.890625 25.203125 \r\n",
       "Q 15.484375 15.921875 20.484375 11.0625 \r\n",
       "Q 25.484375 6.203125 34.421875 6.203125 \r\n",
       "Q 39.59375 6.203125 44.453125 7.46875 \r\n",
       "Q 49.3125 8.734375 54.109375 11.28125 \r\n",
       "L 54.109375 2.78125 \r\n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \r\n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \r\n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \r\n",
       "Q 5.515625 13.8125 5.515625 26.8125 \r\n",
       "Q 5.515625 40.234375 12.765625 48.109375 \r\n",
       "Q 20.015625 56 32.328125 56 \r\n",
       "Q 43.359375 56 49.78125 48.890625 \r\n",
       "Q 56.203125 41.796875 56.203125 29.59375 \r\n",
       "z\r\n",
       "M 47.21875 32.234375 \r\n",
       "Q 47.125 39.59375 43.09375 43.984375 \r\n",
       "Q 39.0625 48.390625 32.421875 48.390625 \r\n",
       "Q 24.90625 48.390625 20.390625 44.140625 \r\n",
       "Q 15.875 39.890625 15.1875 32.171875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-65\"/>\r\n",
       "      <path d=\"M 18.109375 8.203125 \r\n",
       "L 18.109375 -20.796875 \r\n",
       "L 9.078125 -20.796875 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.390625 \r\n",
       "Q 20.953125 51.265625 25.265625 53.625 \r\n",
       "Q 29.59375 56 35.59375 56 \r\n",
       "Q 45.5625 56 51.78125 48.09375 \r\n",
       "Q 58.015625 40.1875 58.015625 27.296875 \r\n",
       "Q 58.015625 14.40625 51.78125 6.484375 \r\n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \r\n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \r\n",
       "Q 20.953125 3.328125 18.109375 8.203125 \r\n",
       "z\r\n",
       "M 48.6875 27.296875 \r\n",
       "Q 48.6875 37.203125 44.609375 42.84375 \r\n",
       "Q 40.53125 48.484375 33.40625 48.484375 \r\n",
       "Q 26.265625 48.484375 22.1875 42.84375 \r\n",
       "Q 18.109375 37.203125 18.109375 27.296875 \r\n",
       "Q 18.109375 17.390625 22.1875 11.75 \r\n",
       "Q 26.265625 6.109375 33.40625 6.109375 \r\n",
       "Q 40.53125 6.109375 44.609375 11.75 \r\n",
       "Q 48.6875 17.390625 48.6875 27.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-70\"/>\r\n",
       "      <path d=\"M 30.609375 48.390625 \r\n",
       "Q 23.390625 48.390625 19.1875 42.75 \r\n",
       "Q 14.984375 37.109375 14.984375 27.296875 \r\n",
       "Q 14.984375 17.484375 19.15625 11.84375 \r\n",
       "Q 23.34375 6.203125 30.609375 6.203125 \r\n",
       "Q 37.796875 6.203125 41.984375 11.859375 \r\n",
       "Q 46.1875 17.53125 46.1875 27.296875 \r\n",
       "Q 46.1875 37.015625 41.984375 42.703125 \r\n",
       "Q 37.796875 48.390625 30.609375 48.390625 \r\n",
       "z\r\n",
       "M 30.609375 56 \r\n",
       "Q 42.328125 56 49.015625 48.375 \r\n",
       "Q 55.71875 40.765625 55.71875 27.296875 \r\n",
       "Q 55.71875 13.875 49.015625 6.21875 \r\n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \r\n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \r\n",
       "Q 5.515625 13.875 5.515625 27.296875 \r\n",
       "Q 5.515625 40.765625 12.171875 48.375 \r\n",
       "Q 18.84375 56 30.609375 56 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6f\"/>\r\n",
       "      <path d=\"M 48.78125 52.59375 \r\n",
       "L 48.78125 44.1875 \r\n",
       "Q 44.96875 46.296875 41.140625 47.34375 \r\n",
       "Q 37.3125 48.390625 33.40625 48.390625 \r\n",
       "Q 24.65625 48.390625 19.8125 42.84375 \r\n",
       "Q 14.984375 37.3125 14.984375 27.296875 \r\n",
       "Q 14.984375 17.28125 19.8125 11.734375 \r\n",
       "Q 24.65625 6.203125 33.40625 6.203125 \r\n",
       "Q 37.3125 6.203125 41.140625 7.25 \r\n",
       "Q 44.96875 8.296875 48.78125 10.40625 \r\n",
       "L 48.78125 2.09375 \r\n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \r\n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \r\n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \r\n",
       "Q 5.515625 14.109375 5.515625 27.296875 \r\n",
       "Q 5.515625 40.671875 12.859375 48.328125 \r\n",
       "Q 20.21875 56 33.015625 56 \r\n",
       "Q 37.15625 56 41.109375 55.140625 \r\n",
       "Q 45.0625 54.296875 48.78125 52.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-63\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 75.984375 \r\n",
       "L 18.109375 75.984375 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-68\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(126.203125 174.876562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-70\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-63\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-68\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m7e6caf0dad\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7e6caf0dad\" y=\"120.202936\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 0.3 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 40.578125 39.3125 \r\n",
       "Q 47.65625 37.796875 51.625 33 \r\n",
       "Q 55.609375 28.21875 55.609375 21.1875 \r\n",
       "Q 55.609375 10.40625 48.1875 4.484375 \r\n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \r\n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \r\n",
       "Q 12.796875 0.390625 7.625 2.203125 \r\n",
       "L 7.625 11.71875 \r\n",
       "Q 11.71875 9.328125 16.59375 8.109375 \r\n",
       "Q 21.484375 6.890625 26.8125 6.890625 \r\n",
       "Q 36.078125 6.890625 40.9375 10.546875 \r\n",
       "Q 45.796875 14.203125 45.796875 21.1875 \r\n",
       "Q 45.796875 27.640625 41.28125 31.265625 \r\n",
       "Q 36.765625 34.90625 28.71875 34.90625 \r\n",
       "L 20.21875 34.90625 \r\n",
       "L 20.21875 43.015625 \r\n",
       "L 29.109375 43.015625 \r\n",
       "Q 36.375 43.015625 40.234375 45.921875 \r\n",
       "Q 44.09375 48.828125 44.09375 54.296875 \r\n",
       "Q 44.09375 59.90625 40.109375 62.90625 \r\n",
       "Q 36.140625 65.921875 28.71875 65.921875 \r\n",
       "Q 24.65625 65.921875 20.015625 65.03125 \r\n",
       "Q 15.375 64.15625 9.8125 62.3125 \r\n",
       "L 9.8125 71.09375 \r\n",
       "Q 15.4375 72.65625 20.34375 73.4375 \r\n",
       "Q 25.25 74.21875 29.59375 74.21875 \r\n",
       "Q 40.828125 74.21875 47.359375 69.109375 \r\n",
       "Q 53.90625 64.015625 53.90625 55.328125 \r\n",
       "Q 53.90625 49.265625 50.4375 45.09375 \r\n",
       "Q 46.96875 40.921875 40.578125 39.3125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-33\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 124.002155)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7e6caf0dad\" y=\"84.585378\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 0.4 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 88.384597)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7e6caf0dad\" y=\"48.967819\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 0.5 -->\r\n",
       "      <g transform=\"translate(20.878125 52.767038)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7e6caf0dad\" y=\"13.350261\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- 0.6 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 33.015625 40.375 \r\n",
       "Q 26.375 40.375 22.484375 35.828125 \r\n",
       "Q 18.609375 31.296875 18.609375 23.390625 \r\n",
       "Q 18.609375 15.53125 22.484375 10.953125 \r\n",
       "Q 26.375 6.390625 33.015625 6.390625 \r\n",
       "Q 39.65625 6.390625 43.53125 10.953125 \r\n",
       "Q 47.40625 15.53125 47.40625 23.390625 \r\n",
       "Q 47.40625 31.296875 43.53125 35.828125 \r\n",
       "Q 39.65625 40.375 33.015625 40.375 \r\n",
       "z\r\n",
       "M 52.59375 71.296875 \r\n",
       "L 52.59375 62.3125 \r\n",
       "Q 48.875 64.0625 45.09375 64.984375 \r\n",
       "Q 41.3125 65.921875 37.59375 65.921875 \r\n",
       "Q 27.828125 65.921875 22.671875 59.328125 \r\n",
       "Q 17.53125 52.734375 16.796875 39.40625 \r\n",
       "Q 19.671875 43.65625 24.015625 45.921875 \r\n",
       "Q 28.375 48.1875 33.59375 48.1875 \r\n",
       "Q 44.578125 48.1875 50.953125 41.515625 \r\n",
       "Q 57.328125 34.859375 57.328125 23.390625 \r\n",
       "Q 57.328125 12.15625 50.6875 5.359375 \r\n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \r\n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \r\n",
       "Q 6.984375 17.96875 6.984375 36.375 \r\n",
       "Q 6.984375 53.65625 15.1875 63.9375 \r\n",
       "Q 23.390625 74.21875 37.203125 74.21875 \r\n",
       "Q 40.921875 74.21875 44.703125 73.484375 \r\n",
       "Q 48.484375 72.75 52.59375 71.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-36\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 17.149479)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_11\">\r\n",
       "     <!-- loss -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6c\"/>\r\n",
       "      <path d=\"M 44.28125 53.078125 \r\n",
       "L 44.28125 44.578125 \r\n",
       "Q 40.484375 46.53125 36.375 47.5 \r\n",
       "Q 32.28125 48.484375 27.875 48.484375 \r\n",
       "Q 21.1875 48.484375 17.84375 46.4375 \r\n",
       "Q 14.5 44.390625 14.5 40.28125 \r\n",
       "Q 14.5 37.15625 16.890625 35.375 \r\n",
       "Q 19.28125 33.59375 26.515625 31.984375 \r\n",
       "L 29.59375 31.296875 \r\n",
       "Q 39.15625 29.25 43.1875 25.515625 \r\n",
       "Q 47.21875 21.78125 47.21875 15.09375 \r\n",
       "Q 47.21875 7.46875 41.1875 3.015625 \r\n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \r\n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \r\n",
       "Q 10.6875 0.296875 5.421875 2 \r\n",
       "L 5.421875 11.28125 \r\n",
       "Q 10.40625 8.6875 15.234375 7.390625 \r\n",
       "Q 20.0625 6.109375 24.8125 6.109375 \r\n",
       "Q 31.15625 6.109375 34.5625 8.28125 \r\n",
       "Q 37.984375 10.453125 37.984375 14.40625 \r\n",
       "Q 37.984375 18.0625 35.515625 20.015625 \r\n",
       "Q 33.0625 21.96875 24.703125 23.78125 \r\n",
       "L 21.578125 24.515625 \r\n",
       "Q 13.234375 26.265625 9.515625 29.90625 \r\n",
       "Q 5.8125 33.546875 5.8125 39.890625 \r\n",
       "Q 5.8125 47.609375 11.28125 51.796875 \r\n",
       "Q 16.75 56 26.8125 56 \r\n",
       "Q 31.78125 56 36.171875 55.265625 \r\n",
       "Q 40.578125 54.546875 44.28125 53.078125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-73\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798437 88.307812)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_10\">\r\n",
       "    <path clip-path=\"url(#pa306b8363a)\" d=\"M 52.658523 16.877273 \r\n",
       "L 58.576705 80.713255 \r\n",
       "L 64.494886 107.484706 \r\n",
       "L 70.413068 118.764375 \r\n",
       "L 76.33125 124.872218 \r\n",
       "L 82.249432 129.756529 \r\n",
       "L 88.167614 134.052745 \r\n",
       "L 94.085795 135.419153 \r\n",
       "L 100.003977 135.577272 \r\n",
       "L 105.922159 134.827322 \r\n",
       "L 111.840341 137.322748 \r\n",
       "L 117.758523 134.877042 \r\n",
       "L 123.676705 138.90027 \r\n",
       "L 129.594886 138.899245 \r\n",
       "L 135.513068 139.529041 \r\n",
       "L 141.43125 139.158057 \r\n",
       "L 147.349432 137.505674 \r\n",
       "L 153.267614 138.809316 \r\n",
       "L 159.185795 139.410935 \r\n",
       "L 165.103977 139.262528 \r\n",
       "L 171.022159 140.183866 \r\n",
       "L 176.940341 140.048203 \r\n",
       "L 182.858523 140.422727 \r\n",
       "L 188.776705 140.259784 \r\n",
       "L 194.694886 138.677554 \r\n",
       "L 200.613068 140.297828 \r\n",
       "L 206.53125 139.063436 \r\n",
       "L 212.449432 139.157441 \r\n",
       "L 218.367614 139.494543 \r\n",
       "L 224.285795 140.088099 \r\n",
       "L 230.203977 139.685722 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 43.78125 146.6 \r\n",
       "L 43.78125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 239.08125 146.6 \r\n",
       "L 239.08125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 43.78125 146.6 \r\n",
       "L 239.08125 146.6 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 43.78125 10.7 \r\n",
       "L 239.08125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"pa306b8363a\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"43.78125\" y=\"10.7\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 例如: optimizer_fn=torch.optim.SGD, optimizer_hyperparams={\"lr\": 0.05}\n",
    "def train_pytorch_ch7(optimizer_fn, optimizer_hyperparams, features, labels,\n",
    "                    batch_size=10, num_epochs=2):\n",
    "    # 初始化模型\n",
    "    net = nn.Sequential(\n",
    "        nn.Linear(features.shape[-1], 1)\n",
    "    )\n",
    "    loss = nn.MSELoss()\n",
    "    optimizer = optimizer_fn(net.parameters(), **optimizer_hyperparams)\n",
    "\n",
    "    def eval_loss():\n",
    "        return loss(net(features).view(-1), labels).item() / 2\n",
    "\n",
    "    ls = [eval_loss()]\n",
    "    data_iter = torch.utils.data.DataLoader(\n",
    "        torch.utils.data.TensorDataset(features, labels), batch_size, shuffle=True)\n",
    "\n",
    "    for _ in range(num_epochs):\n",
    "        start = time.time()\n",
    "        for batch_i, (X, y) in enumerate(data_iter):\n",
    "            # 除以2是为了和train_ch7保持一致, 因为squared_loss中除了2\n",
    "            l = loss(net(X).view(-1), y) / 2 \n",
    "            \n",
    "            optimizer.zero_grad()\n",
    "            l.backward()\n",
    "            optimizer.step()\n",
    "            if (batch_i + 1) * batch_size % 100 == 0:\n",
    "                ls.append(eval_loss())\n",
    "    # 打印结果和作图\n",
    "    print('loss: %f, %f sec per epoch' % (ls[-1], time.time() - start))\n",
    "    dl_utils.set_figsize()\n",
    "    dl_utils.plt.plot(np.linspace(0, num_epochs, len(ls)), ls)\n",
    "    dl_utils.plt.xlabel('epoch')\n",
    "    dl_utils.plt.ylabel('loss')\n",
    "\n",
    "train_pytorch_ch7(optim.SGD, {\"lr\": 0.05}, features, labels, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 小结\n",
    "\n",
    "- 小批量随机梯度每次随机均匀采样一个小批量的训练样本来计算梯度。\n",
    "- 在实际中，（小批量）随机梯度下降的学习率可以在迭代过程中自我衰减。\n",
    "- 通常，小批量随机梯度在每个迭代周期的耗时介于梯度下降和随机梯度下降的耗时之间。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  动量法\n",
    "\n",
    "在7.2节（梯度下降和随机梯度下降）中我们提到，目标函数有关自变量的梯度代表了目标函数在自变量当前位置下降最快的方向。因此，梯度下降也叫作最陡下降（steepest descent）。在每次迭代中，梯度下降根据自变量当前位置，沿着当前位置的梯度更新自变量。然而，如果自变量的迭代方向仅仅取决于自变量当前位置，这可能会带来一些问题。\n",
    "\n",
    "动量法的提出是为了解决梯度下降的上述问题。由于小批量随机梯度下降比梯度下降更为广义，本章后续讨论将沿用7.3节（小批量随机梯度下降）中时间步$t$的小批量随机梯度$\\boldsymbol{g}_t$的定义。设时间步$t$的自变量为$\\boldsymbol{x}_t$，学习率为$\\eta_t$。\n",
    "在时间步$0$，动量法创建速度变量$\\boldsymbol{v}_0$，并将其元素初始化成0。在时间步$t>0$，动量法对每次迭代的步骤做如下修改：\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\boldsymbol{v}_t &\\leftarrow \\gamma \\boldsymbol{v}_{t-1} + \\eta_t \\boldsymbol{g}_t, \\\\\n",
    "\\boldsymbol{x}_t &\\leftarrow \\boldsymbol{x}_{t-1} - \\boldsymbol{v}_t,\n",
    "\\end{aligned}\n",
    "$$\n",
    "其中，动量超参数$\\gamma$满足$0 \\leq \\gamma < 1$。当$\\gamma=0$时，动量法等价于小批量随机梯度下降。\n",
    "\n",
    "### 指数加权移动平均\n",
    "\n",
    "为了从数学上理解动量法，让我们先解释一下指数加权移动平均（exponentially weighted moving average）。给定超参数$0 \\leq \\gamma < 1$，当前时间步$t$的变量$y_t$是上一时间步$t-1$的变量$y_{t-1}$和当前时间步另一变量$x_t$的线性组合：\n",
    "\n",
    "$$y_t = \\gamma y_{t-1} + (1-\\gamma) x_t.$$\n",
    "\n",
    "我们可以对$y_t$展开：\n",
    "$$\n",
    "\\begin{aligned}\n",
    "y_t  &= (1-\\gamma) x_t + \\gamma y_{t-1}\\\\\n",
    "         &= (1-\\gamma)x_t + (1-\\gamma) \\cdot \\gamma x_{t-1} + \\gamma^2y_{t-2}\\\\\n",
    "         &= (1-\\gamma)x_t + (1-\\gamma) \\cdot \\gamma x_{t-1} + (1-\\gamma) \\cdot \\gamma^2x_{t-2} + \\gamma^3y_{t-3}\\\\\n",
    "         &\\ldots\n",
    "\\end{aligned}\n",
    "$$\n",
    "令$n = 1/(1-\\gamma)$，那么 $\\left(1-1/n\\right)^n = \\gamma^{1/(1-\\gamma)}$。因为\n",
    "\n",
    "$$ \\lim_{n \\rightarrow \\infty}  \\left(1-\\frac{1}{n}\\right)^n = \\exp(-1) \\approx 0.3679,$$\n",
    "\n",
    "所以当$\\gamma \\rightarrow 1$时，$\\gamma^{1/(1-\\gamma)}=\\exp(-1)$，如$0.95^{20} \\approx \\exp(-1)$。如果把$\\exp(-1)$当作一个比较小的数，我们可以在近似中忽略所有含$\\gamma^{1/(1-\\gamma)}$和比$\\gamma^{1/(1-\\gamma)}$更高阶的系数的项。例如，当$\\gamma=0.95$时，\n",
    "\n",
    "$$y_t \\approx 0.05 \\sum_{i=0}^{19} 0.95^i x_{t-i}.$$\n",
    "\n",
    "因此，在实际中，我们常常将$y_t$看作是对最近$1/(1-\\gamma)$个时间步的$x_t$值的加权平均。例如，当$\\gamma = 0.95$时，$y_t$可以被看作对最近20个时间步的$x_t$值的加权平均；当$\\gamma = 0.9$时，$y_t$可以看作是对最近10个时间步的$x_t$值的加权平均。而且，离当前时间步$t$越近的$x_t$值获得的权重越大（越接近1）。\n",
    "\n",
    "###  由指数加权移动平均理解动量法\n",
    "\n",
    "现在，我们对动量法的速度变量做变形：\n",
    "\n",
    "$$\\boldsymbol{v}_t \\leftarrow \\gamma \\boldsymbol{v}_{t-1} + (1 - \\gamma) \\left(\\frac{\\eta_t}{1 - \\gamma} \\boldsymbol{g}_t\\right). $$\n",
    "\n",
    "由指数加权移动平均的形式可得，速度变量$\\boldsymbol{v}_t$实际上对序列$\\{\\eta_{t-i}\\boldsymbol{g}_{t-i} /(1-\\gamma):i=0,\\ldots,1/(1-\\gamma)-1\\}$做了指数加权移动平均。换句话说，相比于小批量随机梯度下降，**动量法在每个时间步的自变量更新量近似于将最近$1/(1-\\gamma)$个时间步的普通更新量（即学习率乘以梯度）做了指数加权移动平均后再除以$1-\\gamma$**。所以，在动量法中，自变量在各个方向上的移动幅度不仅取决当前梯度，还取决于过去的各个梯度在各个方向上是否一致。在本节之前示例的优化问题中，所有梯度在水平方向上为正（向右），而在竖直方向上时正（向上）时负（向下）。这样，我们就可以使用较大的学习率，从而使自变量向最优解更快移动。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 0.255635, 0.065848 sec per epoch\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\r\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 249.78125 184.15625\" width=\"249.78125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">\r\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\r\n",
       "  </style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 184.15625 \r\n",
       "L 249.78125 184.15625 \r\n",
       "L 249.78125 -0 \r\n",
       "L 0 -0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 43.78125 146.6 \r\n",
       "L 239.08125 146.6 \r\n",
       "L 239.08125 10.7 \r\n",
       "L 43.78125 10.7 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m965637cd6c\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.658523\" xlink:href=\"#m965637cd6c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 0.0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 31.78125 66.40625 \r\n",
       "Q 24.171875 66.40625 20.328125 58.90625 \r\n",
       "Q 16.5 51.421875 16.5 36.375 \r\n",
       "Q 16.5 21.390625 20.328125 13.890625 \r\n",
       "Q 24.171875 6.390625 31.78125 6.390625 \r\n",
       "Q 39.453125 6.390625 43.28125 13.890625 \r\n",
       "Q 47.125 21.390625 47.125 36.375 \r\n",
       "Q 47.125 51.421875 43.28125 58.90625 \r\n",
       "Q 39.453125 66.40625 31.78125 66.40625 \r\n",
       "z\r\n",
       "M 31.78125 74.21875 \r\n",
       "Q 44.046875 74.21875 50.515625 64.515625 \r\n",
       "Q 56.984375 54.828125 56.984375 36.375 \r\n",
       "Q 56.984375 17.96875 50.515625 8.265625 \r\n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \r\n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \r\n",
       "Q 6.59375 17.96875 6.59375 36.375 \r\n",
       "Q 6.59375 54.828125 13.0625 64.515625 \r\n",
       "Q 19.53125 74.21875 31.78125 74.21875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\"/>\r\n",
       "       <path d=\"M 10.6875 12.40625 \r\n",
       "L 21 12.40625 \r\n",
       "L 21 0 \r\n",
       "L 10.6875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2e\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(44.70696 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"97.044886\" xlink:href=\"#m965637cd6c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 0.5 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 10.796875 72.90625 \r\n",
       "L 49.515625 72.90625 \r\n",
       "L 49.515625 64.59375 \r\n",
       "L 19.828125 64.59375 \r\n",
       "L 19.828125 46.734375 \r\n",
       "Q 21.96875 47.46875 24.109375 47.828125 \r\n",
       "Q 26.265625 48.1875 28.421875 48.1875 \r\n",
       "Q 40.625 48.1875 47.75 41.5 \r\n",
       "Q 54.890625 34.8125 54.890625 23.390625 \r\n",
       "Q 54.890625 11.625 47.5625 5.09375 \r\n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \r\n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \r\n",
       "Q 12.796875 0.140625 7.71875 1.703125 \r\n",
       "L 7.71875 11.625 \r\n",
       "Q 12.109375 9.234375 16.796875 8.0625 \r\n",
       "Q 21.484375 6.890625 26.703125 6.890625 \r\n",
       "Q 35.15625 6.890625 40.078125 11.328125 \r\n",
       "Q 45.015625 15.765625 45.015625 23.390625 \r\n",
       "Q 45.015625 31 40.078125 35.4375 \r\n",
       "Q 35.15625 39.890625 26.703125 39.890625 \r\n",
       "Q 22.75 39.890625 18.8125 39.015625 \r\n",
       "Q 14.890625 38.140625 10.796875 36.28125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-35\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(89.093324 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"141.43125\" xlink:href=\"#m965637cd6c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 1.0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 12.40625 8.296875 \r\n",
       "L 28.515625 8.296875 \r\n",
       "L 28.515625 63.921875 \r\n",
       "L 10.984375 60.40625 \r\n",
       "L 10.984375 69.390625 \r\n",
       "L 28.421875 72.90625 \r\n",
       "L 38.28125 72.90625 \r\n",
       "L 38.28125 8.296875 \r\n",
       "L 54.390625 8.296875 \r\n",
       "L 54.390625 0 \r\n",
       "L 12.40625 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(133.479688 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"185.817614\" xlink:href=\"#m965637cd6c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 1.5 -->\r\n",
       "      <g transform=\"translate(177.866051 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"230.203977\" xlink:href=\"#m965637cd6c\" y=\"146.6\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 2.0 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 19.1875 8.296875 \r\n",
       "L 53.609375 8.296875 \r\n",
       "L 53.609375 0 \r\n",
       "L 7.328125 0 \r\n",
       "L 7.328125 8.296875 \r\n",
       "Q 12.9375 14.109375 22.625 23.890625 \r\n",
       "Q 32.328125 33.6875 34.8125 36.53125 \r\n",
       "Q 39.546875 41.84375 41.421875 45.53125 \r\n",
       "Q 43.3125 49.21875 43.3125 52.78125 \r\n",
       "Q 43.3125 58.59375 39.234375 62.25 \r\n",
       "Q 35.15625 65.921875 28.609375 65.921875 \r\n",
       "Q 23.96875 65.921875 18.8125 64.3125 \r\n",
       "Q 13.671875 62.703125 7.8125 59.421875 \r\n",
       "L 7.8125 69.390625 \r\n",
       "Q 13.765625 71.78125 18.9375 73 \r\n",
       "Q 24.125 74.21875 28.421875 74.21875 \r\n",
       "Q 39.75 74.21875 46.484375 68.546875 \r\n",
       "Q 53.21875 62.890625 53.21875 53.421875 \r\n",
       "Q 53.21875 48.921875 51.53125 44.890625 \r\n",
       "Q 49.859375 40.875 45.40625 35.40625 \r\n",
       "Q 44.1875 33.984375 37.640625 27.21875 \r\n",
       "Q 31.109375 20.453125 19.1875 8.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(222.252415 161.198437)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_6\">\r\n",
       "     <!-- epoch -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 56.203125 29.59375 \r\n",
       "L 56.203125 25.203125 \r\n",
       "L 14.890625 25.203125 \r\n",
       "Q 15.484375 15.921875 20.484375 11.0625 \r\n",
       "Q 25.484375 6.203125 34.421875 6.203125 \r\n",
       "Q 39.59375 6.203125 44.453125 7.46875 \r\n",
       "Q 49.3125 8.734375 54.109375 11.28125 \r\n",
       "L 54.109375 2.78125 \r\n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \r\n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \r\n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \r\n",
       "Q 5.515625 13.8125 5.515625 26.8125 \r\n",
       "Q 5.515625 40.234375 12.765625 48.109375 \r\n",
       "Q 20.015625 56 32.328125 56 \r\n",
       "Q 43.359375 56 49.78125 48.890625 \r\n",
       "Q 56.203125 41.796875 56.203125 29.59375 \r\n",
       "z\r\n",
       "M 47.21875 32.234375 \r\n",
       "Q 47.125 39.59375 43.09375 43.984375 \r\n",
       "Q 39.0625 48.390625 32.421875 48.390625 \r\n",
       "Q 24.90625 48.390625 20.390625 44.140625 \r\n",
       "Q 15.875 39.890625 15.1875 32.171875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-65\"/>\r\n",
       "      <path d=\"M 18.109375 8.203125 \r\n",
       "L 18.109375 -20.796875 \r\n",
       "L 9.078125 -20.796875 \r\n",
       "L 9.078125 54.6875 \r\n",
       "L 18.109375 54.6875 \r\n",
       "L 18.109375 46.390625 \r\n",
       "Q 20.953125 51.265625 25.265625 53.625 \r\n",
       "Q 29.59375 56 35.59375 56 \r\n",
       "Q 45.5625 56 51.78125 48.09375 \r\n",
       "Q 58.015625 40.1875 58.015625 27.296875 \r\n",
       "Q 58.015625 14.40625 51.78125 6.484375 \r\n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \r\n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \r\n",
       "Q 20.953125 3.328125 18.109375 8.203125 \r\n",
       "z\r\n",
       "M 48.6875 27.296875 \r\n",
       "Q 48.6875 37.203125 44.609375 42.84375 \r\n",
       "Q 40.53125 48.484375 33.40625 48.484375 \r\n",
       "Q 26.265625 48.484375 22.1875 42.84375 \r\n",
       "Q 18.109375 37.203125 18.109375 27.296875 \r\n",
       "Q 18.109375 17.390625 22.1875 11.75 \r\n",
       "Q 26.265625 6.109375 33.40625 6.109375 \r\n",
       "Q 40.53125 6.109375 44.609375 11.75 \r\n",
       "Q 48.6875 17.390625 48.6875 27.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-70\"/>\r\n",
       "      <path d=\"M 30.609375 48.390625 \r\n",
       "Q 23.390625 48.390625 19.1875 42.75 \r\n",
       "Q 14.984375 37.109375 14.984375 27.296875 \r\n",
       "Q 14.984375 17.484375 19.15625 11.84375 \r\n",
       "Q 23.34375 6.203125 30.609375 6.203125 \r\n",
       "Q 37.796875 6.203125 41.984375 11.859375 \r\n",
       "Q 46.1875 17.53125 46.1875 27.296875 \r\n",
       "Q 46.1875 37.015625 41.984375 42.703125 \r\n",
       "Q 37.796875 48.390625 30.609375 48.390625 \r\n",
       "z\r\n",
       "M 30.609375 56 \r\n",
       "Q 42.328125 56 49.015625 48.375 \r\n",
       "Q 55.71875 40.765625 55.71875 27.296875 \r\n",
       "Q 55.71875 13.875 49.015625 6.21875 \r\n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \r\n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \r\n",
       "Q 5.515625 13.875 5.515625 27.296875 \r\n",
       "Q 5.515625 40.765625 12.171875 48.375 \r\n",
       "Q 18.84375 56 30.609375 56 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6f\"/>\r\n",
       "      <path d=\"M 48.78125 52.59375 \r\n",
       "L 48.78125 44.1875 \r\n",
       "Q 44.96875 46.296875 41.140625 47.34375 \r\n",
       "Q 37.3125 48.390625 33.40625 48.390625 \r\n",
       "Q 24.65625 48.390625 19.8125 42.84375 \r\n",
       "Q 14.984375 37.3125 14.984375 27.296875 \r\n",
       "Q 14.984375 17.28125 19.8125 11.734375 \r\n",
       "Q 24.65625 6.203125 33.40625 6.203125 \r\n",
       "Q 37.3125 6.203125 41.140625 7.25 \r\n",
       "Q 44.96875 8.296875 48.78125 10.40625 \r\n",
       "L 48.78125 2.09375 \r\n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \r\n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \r\n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \r\n",
       "Q 5.515625 14.109375 5.515625 27.296875 \r\n",
       "Q 5.515625 40.671875 12.859375 48.328125 \r\n",
       "Q 20.21875 56 33.015625 56 \r\n",
       "Q 37.15625 56 41.109375 55.140625 \r\n",
       "Q 45.0625 54.296875 48.78125 52.59375 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-63\"/>\r\n",
       "      <path d=\"M 54.890625 33.015625 \r\n",
       "L 54.890625 0 \r\n",
       "L 45.90625 0 \r\n",
       "L 45.90625 32.71875 \r\n",
       "Q 45.90625 40.484375 42.875 44.328125 \r\n",
       "Q 39.84375 48.1875 33.796875 48.1875 \r\n",
       "Q 26.515625 48.1875 22.3125 43.546875 \r\n",
       "Q 18.109375 38.921875 18.109375 30.90625 \r\n",
       "L 18.109375 0 \r\n",
       "L 9.078125 0 \r\n",
       "L 9.078125 75.984375 \r\n",
       "L 18.109375 75.984375 \r\n",
       "L 18.109375 46.1875 \r\n",
       "Q 21.34375 51.125 25.703125 53.5625 \r\n",
       "Q 30.078125 56 35.796875 56 \r\n",
       "Q 45.21875 56 50.046875 50.171875 \r\n",
       "Q 54.890625 44.34375 54.890625 33.015625 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-68\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(126.203125 174.876562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-70\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-63\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-68\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"ma6468dc890\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#ma6468dc890\" y=\"120.352454\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 0.3 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 40.578125 39.3125 \r\n",
       "Q 47.65625 37.796875 51.625 33 \r\n",
       "Q 55.609375 28.21875 55.609375 21.1875 \r\n",
       "Q 55.609375 10.40625 48.1875 4.484375 \r\n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \r\n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \r\n",
       "Q 12.796875 0.390625 7.625 2.203125 \r\n",
       "L 7.625 11.71875 \r\n",
       "Q 11.71875 9.328125 16.59375 8.109375 \r\n",
       "Q 21.484375 6.890625 26.8125 6.890625 \r\n",
       "Q 36.078125 6.890625 40.9375 10.546875 \r\n",
       "Q 45.796875 14.203125 45.796875 21.1875 \r\n",
       "Q 45.796875 27.640625 41.28125 31.265625 \r\n",
       "Q 36.765625 34.90625 28.71875 34.90625 \r\n",
       "L 20.21875 34.90625 \r\n",
       "L 20.21875 43.015625 \r\n",
       "L 29.109375 43.015625 \r\n",
       "Q 36.375 43.015625 40.234375 45.921875 \r\n",
       "Q 44.09375 48.828125 44.09375 54.296875 \r\n",
       "Q 44.09375 59.90625 40.109375 62.90625 \r\n",
       "Q 36.140625 65.921875 28.71875 65.921875 \r\n",
       "Q 24.65625 65.921875 20.015625 65.03125 \r\n",
       "Q 15.375 64.15625 9.8125 62.3125 \r\n",
       "L 9.8125 71.09375 \r\n",
       "Q 15.4375 72.65625 20.34375 73.4375 \r\n",
       "Q 25.25 74.21875 29.59375 74.21875 \r\n",
       "Q 40.828125 74.21875 47.359375 69.109375 \r\n",
       "Q 53.90625 64.015625 53.90625 55.328125 \r\n",
       "Q 53.90625 49.265625 50.4375 45.09375 \r\n",
       "Q 46.96875 40.921875 40.578125 39.3125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-33\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 124.151672)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#ma6468dc890\" y=\"84.45306\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 0.4 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 37.796875 64.3125 \r\n",
       "L 12.890625 25.390625 \r\n",
       "L 37.796875 25.390625 \r\n",
       "z\r\n",
       "M 35.203125 72.90625 \r\n",
       "L 47.609375 72.90625 \r\n",
       "L 47.609375 25.390625 \r\n",
       "L 58.015625 25.390625 \r\n",
       "L 58.015625 17.1875 \r\n",
       "L 47.609375 17.1875 \r\n",
       "L 47.609375 0 \r\n",
       "L 37.796875 0 \r\n",
       "L 37.796875 17.1875 \r\n",
       "L 4.890625 17.1875 \r\n",
       "L 4.890625 26.703125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 88.252279)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#ma6468dc890\" y=\"48.553666\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 0.5 -->\r\n",
       "      <g transform=\"translate(20.878125 52.352885)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#ma6468dc890\" y=\"12.654273\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- 0.6 -->\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 33.015625 40.375 \r\n",
       "Q 26.375 40.375 22.484375 35.828125 \r\n",
       "Q 18.609375 31.296875 18.609375 23.390625 \r\n",
       "Q 18.609375 15.53125 22.484375 10.953125 \r\n",
       "Q 26.375 6.390625 33.015625 6.390625 \r\n",
       "Q 39.65625 6.390625 43.53125 10.953125 \r\n",
       "Q 47.40625 15.53125 47.40625 23.390625 \r\n",
       "Q 47.40625 31.296875 43.53125 35.828125 \r\n",
       "Q 39.65625 40.375 33.015625 40.375 \r\n",
       "z\r\n",
       "M 52.59375 71.296875 \r\n",
       "L 52.59375 62.3125 \r\n",
       "Q 48.875 64.0625 45.09375 64.984375 \r\n",
       "Q 41.3125 65.921875 37.59375 65.921875 \r\n",
       "Q 27.828125 65.921875 22.671875 59.328125 \r\n",
       "Q 17.53125 52.734375 16.796875 39.40625 \r\n",
       "Q 19.671875 43.65625 24.015625 45.921875 \r\n",
       "Q 28.375 48.1875 33.59375 48.1875 \r\n",
       "Q 44.578125 48.1875 50.953125 41.515625 \r\n",
       "Q 57.328125 34.859375 57.328125 23.390625 \r\n",
       "Q 57.328125 12.15625 50.6875 5.359375 \r\n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \r\n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \r\n",
       "Q 6.984375 17.96875 6.984375 36.375 \r\n",
       "Q 6.984375 53.65625 15.1875 63.9375 \r\n",
       "Q 23.390625 74.21875 37.203125 74.21875 \r\n",
       "Q 40.921875 74.21875 44.703125 73.484375 \r\n",
       "Q 48.484375 72.75 52.59375 71.296875 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-36\"/>\r\n",
       "      </defs>\r\n",
       "      <g transform=\"translate(20.878125 16.453492)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_11\">\r\n",
       "     <!-- loss -->\r\n",
       "     <defs>\r\n",
       "      <path d=\"M 9.421875 75.984375 \r\n",
       "L 18.40625 75.984375 \r\n",
       "L 18.40625 0 \r\n",
       "L 9.421875 0 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6c\"/>\r\n",
       "      <path d=\"M 44.28125 53.078125 \r\n",
       "L 44.28125 44.578125 \r\n",
       "Q 40.484375 46.53125 36.375 47.5 \r\n",
       "Q 32.28125 48.484375 27.875 48.484375 \r\n",
       "Q 21.1875 48.484375 17.84375 46.4375 \r\n",
       "Q 14.5 44.390625 14.5 40.28125 \r\n",
       "Q 14.5 37.15625 16.890625 35.375 \r\n",
       "Q 19.28125 33.59375 26.515625 31.984375 \r\n",
       "L 29.59375 31.296875 \r\n",
       "Q 39.15625 29.25 43.1875 25.515625 \r\n",
       "Q 47.21875 21.78125 47.21875 15.09375 \r\n",
       "Q 47.21875 7.46875 41.1875 3.015625 \r\n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \r\n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \r\n",
       "Q 10.6875 0.296875 5.421875 2 \r\n",
       "L 5.421875 11.28125 \r\n",
       "Q 10.40625 8.6875 15.234375 7.390625 \r\n",
       "Q 20.0625 6.109375 24.8125 6.109375 \r\n",
       "Q 31.15625 6.109375 34.5625 8.28125 \r\n",
       "Q 37.984375 10.453125 37.984375 14.40625 \r\n",
       "Q 37.984375 18.0625 35.515625 20.015625 \r\n",
       "Q 33.0625 21.96875 24.703125 23.78125 \r\n",
       "L 21.578125 24.515625 \r\n",
       "Q 13.234375 26.265625 9.515625 29.90625 \r\n",
       "Q 5.8125 33.546875 5.8125 39.890625 \r\n",
       "Q 5.8125 47.609375 11.28125 51.796875 \r\n",
       "Q 16.75 56 26.8125 56 \r\n",
       "Q 31.78125 56 36.171875 55.265625 \r\n",
       "Q 40.578125 54.546875 44.28125 53.078125 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-73\"/>\r\n",
       "     </defs>\r\n",
       "     <g transform=\"translate(14.798437 88.307812)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_10\">\r\n",
       "    <path clip-path=\"url(#p1ef986ee6d)\" d=\"M 52.658523 16.877273 \r\n",
       "L 58.576705 43.261286 \r\n",
       "L 64.494886 79.651669 \r\n",
       "L 70.413068 107.143934 \r\n",
       "L 76.33125 121.342264 \r\n",
       "L 82.249432 127.573405 \r\n",
       "L 88.167614 131.45941 \r\n",
       "L 94.085795 132.365935 \r\n",
       "L 100.003977 134.254742 \r\n",
       "L 105.922159 135.136649 \r\n",
       "L 111.840341 136.044286 \r\n",
       "L 117.758523 136.438956 \r\n",
       "L 123.676705 136.683425 \r\n",
       "L 129.594886 138.247939 \r\n",
       "L 135.513068 138.67308 \r\n",
       "L 141.43125 138.015196 \r\n",
       "L 147.349432 138.34845 \r\n",
       "L 153.267614 138.955529 \r\n",
       "L 159.185795 139.63802 \r\n",
       "L 165.103977 138.485133 \r\n",
       "L 171.022159 139.44428 \r\n",
       "L 176.940341 140.264363 \r\n",
       "L 182.858523 139.363921 \r\n",
       "L 188.776705 139.024034 \r\n",
       "L 194.694886 137.706502 \r\n",
       "L 200.613068 136.697686 \r\n",
       "L 206.53125 138.241734 \r\n",
       "L 212.449432 140.422727 \r\n",
       "L 218.367614 139.701929 \r\n",
       "L 224.285795 137.909149 \r\n",
       "L 230.203977 136.279083 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 43.78125 146.6 \r\n",
       "L 43.78125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 239.08125 146.6 \r\n",
       "L 239.08125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 43.78125 146.6 \r\n",
       "L 239.08125 146.6 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 43.78125 10.7 \r\n",
       "L 239.08125 10.7 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p1ef986ee6d\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"43.78125\" y=\"10.7\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_pytorch_ch7(torch.optim.SGD, {'lr': 0.004, 'momentum': 0.9},\n",
    "                    features, labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 小结\n",
    "\n",
    "- 动量法使用了指数加权移动平均的思想。它将过去时间步的梯度做了加权平均，且权重按时间步指数衰减。\n",
    "- 动量法使得相邻时间步的自变量更新在方向上更加一致。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AdaGrad算法\n",
    "RMSProp算法   \n",
    "AdaDelta算法  \n",
    "Adam算法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
